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CHAPTER I 


Introduction: Why We Need Critical 
Perspectives on Al 


Pieter Verdegem 


Introduction 


Ihe renewed interest in Artificial Intelligence (AI) has made it the most 
recent hype in the world of technological innovation. In the business world, 
Al is seen as a catalyst for growth, which will manifestly transform the eco- 
nomy and the future of work (Agrawal, Gans and Goldfarb 2018; Lee 2018; 
McAfee and Brynjolfsson 2017). Policymakers and civil society are putting 
their hopes on AI for tackling global challenges such as pandemics and even 
climate change (Dobbe and Whittaker 2019; Dananjayan and Raj 2020). Al also 
seems to be the subject of an arms race between China, Russia and the USA for 
equipping their armies with automated weaponry (Asaro 2018). 

Whenever we are confronted with a hype, it is of utmost importance to untan- 
gle what exactly is at stake and who is behind the discourses and myths created. 
We are being told stories about Al as the ultimate innovation, transforming the 
ways we live and work - often started in corporate circles and distributed by 
their supportive popular outlets. At the same time, however, analysis is reveal- 
ing that AI itself is one reason behind intensifying societal problems and harms. 
Researchers and thinkers have observed and/or predicted that AI leads to dis- 
crimination (Zuiderveen Borgesius 2018), is the engine behind growing inequal- 
ities (Korinek and Stiglitz 2017), can bring about technological unemployment 
(Ford 2015) and may even contribute to the end of humanity (Bostrom 2014). 
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Amidst this doom and gloom, what we desperately need is a more nuanced 
debate about AI’s risks and opportunities. This can - must - be a serious and 
informed discussion that goes beyond hyperbole and polarisation, fuelled by 
popular media and thus feeding into public debate. What we need is critical 
perspectives on AI: what it is and what it is not; what type of AI we need, what 
visions exist about this and who is behind them; and ultimately, how to think 
and talk about AI power and inequalities. 

In one word, it is power that must be at the centre of our conversations about 
AI and that is what this book is about. If we want to talk about critical perspec- 
tives on AI, formulating a critique on AI, how it is currently being developed 
and discussed, and yes, if we are serious about making sure that AI will benefit 
everyone, we need to talk about power. Power refers to the capacity to influ- 
ence the actions, beliefs or behaviour of others. Ultimately, this comes down 
to ‘the question of who can influence what society looks like and who controls 
the means that allow such influence’ (Fuchs 2017: 86). Power decides who will 
benefit from new technologies such as AI, but a concentration of power 
will likely result in growing inequalities and other negative outcomes. The 
current critiques about AI centre on AI ethics (Coeckelbergh 2020), which is 
valuable and important to shape policy discussions. AI ethics, however, also 
has serious limitations when it comes to bringing about real change and mak- 
ing sure that the benefits of AI are accessible for everyone. Further in this intro- 
duction, I elaborate on this and I make the case for a radical democratisation of 
AI, and why we need to put power at the centre for achieving this. 

The contributions in this book braid discussion of power and critique with 
three strands: AI - Humans vs. Machines, Discourses and Myths About AI and 
AI Power and Inequalities. 

Part 1: AI - Humans vs. Machines - deals with the history and conceptualisa- 
tion of AI and what is at stake in its development. This section looks at different 
perspectives about what characterises machine intelligence and how it might 
be important to further radical humanism in the era of automation and Al. 

Part 2: Discourses and Myths About AI - analyses how AI is framed in popu- 
lar and scholarly discussions and investigates the normative projections of what 
AI should be and what it should do. This section poses critical questions about 
how AI needs to debunk the myths surrounding it. 

Part 3: AI Power and Inequalities - advances the debate around AI by criti- 
cally examining what ‘AI for Everyone?’ means. This is dealing with the root of 
the problem: who will benefit from AI is ultimately down to who has the power 
to decide. These contributions look at how AI capitalism is organised, what 
(new) inequalities it might bring about and how we can fight back. 

Why do we need a book on AI for Everyone? and why do we need it now? 
The 2007-2008 financial crisis, and the resulting global economic crisis, has not 
only brought about a decade of austerity in large parts of the Western world; 
it has also been the context in which social media and digital platforms have 
transformed into behemoths. Tech companies are now dominating the top 10 
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of the most valuable companies in the world (Verdegem 2021). Austerity has 
also led to growing inequalities and political polarisation, bringing right-wing 
authoritarian politics into power in a number of countries (Fuchs 2018). A 
world already cracked by economic uncertainty and the looming threat of 
climate change was then shaken in 2020 by a global pandemic. COVID-19 
has massively impacted the global economy, on a much larger scale than the 
2007-2008 crisis. On top of this, the pandemic has also resulted in an even 
bigger dependence and dominance of tech platforms such as Amazon, Alibaba, 
Google and Tencent. These companies are, not surprisingly, also leading AI 
companies. Only a small number of corporations have the necessary compu- 
tational power to develop AI systems, are financially strong enough to hire the 
brightest AI talent and have access to the gigantic datasets that are needed to 
train machine learning and deep learning (AI) models. This context makes it 
very clear why we need to ask critical questions about AI and power. 


Conceptualising AI - What AI Are We Talking About? 


Before understanding what type of AI we want, we need to understand what AI 
we have. This is an area of significant debate, and the book opens by exploring 
the varying approaches to how we define AI. 


The Origins of AI 


It is easy to forget that AI has been with us for more than 60 years. Despite 
the flash of excitement and anxiety that feels so recent, AI itself is not a new 
phenomenon. The name Artificial Intelligence (AI) was coined in the mid- 
1950s at a series of academic workshops organised at Dartmouth College, New 
Hampshire (USA). A group of scientists, led by mathematics professor John 
McCarthy, gathered to investigate the ways in which machines could simulate 
aspects of human intelligence: the ability to learn and make decisions. Their 
core assumption was that human reasoning could be reconstructed using 
mathematical techniques and, as a consequence, problem-solving could be 
formalised into algorithms (McCarthy et al. 1955/2006). 

What is more recent is a reflexive, if not critical, and social-scientific, under- 
standing of not just ATs capabilities, but its impacts on human life and social 
organisation (Elliott 2019). It took decades for AI research to move from 
what it could do for us to what it could do to us, or enable us to do to each 
other. These first critical insights came along with observations that AI can 
not only supercharge innovation and bring about economic prosperity but also 
lead to inequalities and unfairness. 

This book contributes to this debate by critically reflecting on how we 
should think about AI and the relationship between humans and machines. 
It analyses the discourses and myths that exist around AJ; what it will enable 
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and what not. And it looks at issues about AI, power and inequalities, investi- 
gating where the risks of exclusion are and how we should deal with this. 

The book also brings diverse and critical voices to this debate. Whereas AI 
as a discipline has been dominated by white, male, predominantly older scien- 
tists from mathematical disciplines, this collection brings perspectives that are 
characterised by a strong diversity in authorship and discipline. And threading 
through all, the contributions offer a discussion of different tangents of power 
and political economy in the field of AI and society. 

The first task is to name our terms. For a concept that has been with us for so 
long, there is little consensus on how to define it. The history of debating AI is 
almost as old as AI itself. There is more debate than agreement about what AI 
is and what it is not, and the only thing generally agreed is that there is no 
widely accepted definition (Russell and Norvig 2016). The first definition comes 
from that gathering of scientists in 1955: McCarthy et al. (1955/2006) then 
defined AI as: ‘Making a machine behave in ways that would be called intelli- 
gent if a human were so behaving: This only raises the challenge of how exactly 
to define intelligence. Russell and Norvig (2016: 2) define different approaches to 
AI to serve different goals. AI can refer to systems that: (1) think like humans; 
(2) think rationally; (3) act like humans; and (4) act rationally. Each of the 
approaches requires different disciplinary expertise, thus requiring an inter-, 
or at least cross-disciplinary discussion. The human-centred approaches will 
depart from social science studying human behaviour, while the rationalist 
approaches will involve a combination of mathematics and engineering. From 
the four approaches, acting like humans is closest to how we define and under- 
stand contemporary AI. 

We can see the roots of acting like humans in the Turing test, developed by 
Alan Turing in 1950. This test, originally designed to provide a satisfactory 
definition of intelligence, has been central to conceptualising AI. According to 
the test, if a human interrogator cannot distinguish a machine from a human 
through conversation, then the machine can be considered intelligent. Russell 
and Norvig (2016) argue that for a computer to be intelligent - to pass the 
Turing test - it needs to possess the following capabilities: natural language 
processing (being able to communicate successfully), knowledge representation 
(being able to store what it knows or hears), automated reasoning (being able 
to use the stored information to answer questions and to draw new conclu- 
sions) and machine learning (being able to adapt to new circumstances and to 
detect and extrapolate patterns). 


Towards an Operational Definition - For Now 


It is helpful to first distinguish between strong and weak AI (Bostrom 2014). 
Strong AI, also called AGI (Artificial General Intelligence) refers to com- 
putational systems with general cognitive abilities which have the future 
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potential to surpass human intellectual capacities. This can be seen as the attempt 
to mechanise human-level intelligence. Computer scientists and philosophers 
disagree on whether this is at all possible (Coeckelbergh 2020): some directly 
reject this scenario while others think if theoretically possible, it is not likely to 
happen (soon) in practice (Boden 2016). This is why it might be better to focus 
on advancements in weak AI or ANI (Artificial Narrow Intelligence), as this is 
the type of AI already impacting everyday life on a massive scale. Weak/nar- 
row AI performs specific tasks which would normally require intelligence in a 
human being - machines aiding human thought and action. This type of AI is a 
mathematical method for prediction (Agrawal et al. 2018). Such systems can be 
extremely powerful but are limited in the range of tasks they can perform. 

Russell and Norvig (2016) see machine learning as a prerequisite for intel- 
ligent machines. Machine learning is a paradigm that allows programs to auto- 
matically improve their performance on a particular task by learning from vast 
amounts of data (Alpaydin 2016). It seeks and uses statistical patterns and cor- 
relation in enormous datasets. Unlike older types of AI (e.g. expert systems, 
that are based on rules which are inputted by humans), machine learning algo- 
rithms learn not from humans but from data. The availability of significant 
amounts of real-world data (that we produce by using the internet, social media, 
sensors or other Internet-of-Things applications), combined with the availabil- 
ity of powerful and almost limitless computing capacity and advancements in 
machine learning and deep learning is why we are currently in another period 
of AI optimism and hype (Elish and boyd 2018). 

Given the concepts and the brief discussion above, how can we agree on an 
operational definition of AI? A basic definition would be to refer to AI as com- 
puter programming that learns from and adapts to data. A more elaborate ver- 
sion of this, as Elliott (2019: 4) puts it, defines AI as ‘any computational system 
that can sense its relevant context and react intelligently to data in order to 
perform highly complex tasks effectively and to achieve specific goals, thereby 
mimicking intelligent human behaviours. The discussion about how to define 
AI cannot be settled in one definition, let alone one book. It is an important 
starting point, however, and Part 1 and Part 2 of this book will unpack several 
approaches to defining AI. 


The Realities of AI for Some vs. the Ideals of AI for Everyone 
Visions of AI in Policies and Ethics 


Examining AI policies and ethics helps us to explore questions of what type of 
AI we want/need, how its development should look like and how we deal with 
its impact. Policy development happens at several levels and includes a number 
of stakeholders: national governments, intergovernmental organisations, cor- 
porations, professional associations and academics. 
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While AI policies reflect the priorities of the stakeholders involved, ethi- 
cal guidelines project a vision of what type of AI is preferred, what benefits it 
should deliver and how we should deal with potential risks. Obviously, this is 
part of a normative debate but we can learn a lot from who is involved in these 
discussions and how they aim to shape the future of AI. 

Given the projections about the role of AI in economic development, AI is 
high on the policy agenda. Putin famously said that the nation that leads in AI 
would be the ruler of the world (Vincent 2017). Major nations are rushing to 
create Al initiatives, unsurprisingly led by China and the USA (Lee 2018). What 
is surprising, however, is how much overlap there is in their strategic vision. 

Chinas national strategy for AI, the New Generation Artificial Intelligence 
Development Plan (NGAIDP), was released in 2017 (State Council of China 
2017). China wants to become the world leader in AI by 2030 and has for- 
mulated strategic goals to achieve this, such as making China the superpower 
of fundamental and applied AI research and development in order to domi- 
nate the global AI market. The main focus of China's AI policy is on economic 
development and competition, even though it also discusses some concerns in 
terms of economic security and social stability. 

The Trump administration launched the American AI Initiative in 2019 
(White House 2019). This strategic policy is all about a nationalist vision of 
American leadership in AI. The US government wants to invest in AI R&D, set 
AI standards and build the AI workforce. The Trump AI strategy not surpris- 
ingly has an intense national focus, highlighting AI for American innovation, 
industry, workers and values, aimed at promoting and protecting national AI 
technology and innovation. There is some discussion of public trust and con- 
fidence in AI as well as the protection of civil liberties, privacy and American 
values but this is subordinated to leadership and protecting American AI tech- 
nology. With the election of Biden, it remains to be seen what the shift in AI 
policies will be, but given his track record the US will continue to pursue US 
capitalist interests, although maybe in a less outspoken nationalist way. 

Most European nations where we see AI policy development, including 
France, Germany and the UK, are taking a different approach, which more 
explicitly offers a normative vision of how AI should contribute to social pro- 
gress. France, for example, has entitled its vision AI for Humanity and aims 
for the development of an ethical framework for transparent and fair use of 
AI applications (Villani 2018). Germany also wants to guarantee responsible 
development and deployment of AI which serves the good of society. The UK 
sits somewhere between the continental European visions and the US vision, 
with goals contributing to global AI development, tempered with nationalist 
objectives focusing on specific benefits for the UK. 

It is clear that China and the US are in an intense battle for global AI leader- 
ship and their policies are dominated by nationalist goals. European countries 
want to engage in AI innovation and boost their competitiveness while also 
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ensuring that the societal impact of AI is not forgotten. But still, this does not 
tell us a lot about what type of AI we want/need; it rather explains what coun- 
tries expect Alto do for them. The European Union, however, has done more to 
develop a vision of what type of AI needs to be pursued and what aspects need 
to be dealt with in this. 

The EU situates itself between China (state capitalism) and the US (market 
capitalism) and seeks to shift the debate in terms of the impact on society and 
its citizens. This positioning is aligned with how they have approached Gen- 
eral Data Protection Regulation (GDPR) in the context of data protection and 
privacy. The EU has put forward trustworthy AI as the key term highlighting 
what type of AI it likes to see developed. This concept is the result of an open 
consultation and its ethics guidelines have been presented by the High-Level 
Expert Group of AI. According to these guidelines, trustworthy AI should be: 
(1) lawful (respecting all applicable laws and regulations); (2) ethical (respect 
ethical principles and values); and (3) robust (both from a technical perspec- 
tive while taking into account its social environment) (European Commission 
2019). These aspects are vague (how is something ethical or robust exactly?) 
as well as self-evident (very few people would favour unlawful AI). The EU, 
however, has made these guidelines more explicit by formulating specific aims: 
human agency/oversight, technical robustness/safety, privacy/data governance, 
transparency, diversity/non-discrimination/fairness, societal/environmental 
well-being and accountability. This is helpful as the list of specific aims can be 
read as values we would like to attribute to AI. 

Not only governments or governmental organisations are active in putting 
forward a vision for AI. Companies also have a stake in this debate so it is 
instructive to examine how leading tech companies talk about what type of 
AI they want to build. Google (2020) has developed a vision it calls Advanc- 
ing AI for Everyone, which can be summed up as applying AI to improve their 
products and developing tools to ensure that everyone can access AI. Google 
also has an AI for Social Good project, similar to Microsofts AI for Good pro- 
gram. The latter aims to use AI expertise to solve humanitarian and environ- 
mental challenges: AI for earth, health, accessibility, humanitarian action and 
cultural heritage. While seemingly well-intended at first glance, these AI pro- 
grams are carefully designed to support goals of corporate social responsibility 
(Sandoval 2014) and are undeniably textbook examples of what Morozov (2013) 
has called techno-solutionism. The problem with these corporate AI visions is 
that they lack substance and therefore do not reveal anything about what they 
intend specifically and how they actually can and should benefit society. 

More substance can be found in how professional associations propose a 
vision of what Good AI exactly means. Organisations such as the Institute of 
Electrical and Electronics Engineers (IEEE) and Association for Computing 
Machinery (ACM) have produced codes that propose ethical principles for 
computer science in general and AI in particular. ACM (2020), for example, 
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talks about AI that needs to ‘contribute to society and to human well-being, 
while IEEE (2020) has come up with principles for ethically aligned design. 
General principles include human rights, well-being, data agency, effectiveness, 
transparency, accountability, awareness of misuse and competence. 

Often cited are the Asilomar AI Principles. The Asilomar Conference on Benefi- 
cial AI was organised by the Future of Life Institute (2017) and brought together 
more than 100 AI researchers from academia and industry and thought leaders 
in economics, law, ethics and philosophy to address and formulate principles 
of beneficial AI. The resulting Asilomar AI principles are organised around 
(1) research issues, (2) ethics and values and (3) longer-term issues (Future of 
Life Institute 2017). The first category, research issues, sets out some guidelines 
in terms of research goals, funding and culture. Secondly, thirteen specific eth- 
ics and values are listed, dealing with transparency, safety, privacy, etc. and they 
also address aspects such as shared benefit (‘AI technologies should benefit and 
empower as many people as possible’) and shared prosperity (‘the economic 
prosperity created by AI should be shared broadly, to benefit all of human- 
ity’). Last, under longer-term issues, cautionary aspects and risks are addressed, 
including the principle of common good, which states: ‘Superintelligence [Arti- 
ficial General Intelligence, as discussed above] should only be developed in the 
service of widely shared ethical ideals, and for the benefit of all humanity rather 
than one state or organisation’ 

While the Asilomar AI principles are valid, they leave unclear who can and 
should take ownership and what mechanisms can be developed to enforce 
them. One specific concern of the Asilomar Al initiative is the heavy involve- 
ment of corporate stakeholders, given that it is backed by tech giants includ- 
ing Google, Facebook and Apple. These are not non-profit organisations but 
companies that are among the most wealthy and profitable organisations in the 
world. They might say they want to develop AI applications that are beneficial 
for society but can we trust them not to use their power to shift the direction 
of AI development to their corporate benefit and the return on investment for 
their investors and shareholders? 

The Al4People initiative, set up by the non-profit organisation Atomium- 
EISMD (European Institute for Science, Media and Democracy), is the Euro- 
pean response to the Asilomar AI initiative. Al4People also brings together 
academics, business partners (e.g. Facebook, Intel and Microsoft), and civil 
society organisations. The ambition of Al4People (Atomium-EISMD, 2020) is 
‘to draft a set of ethical guidelines aimed at facilitating the design of policies 
favourable to the development of a “Good AI Society”. 

Al4People has developed an ethical framework of principles that should 
underpin the adoption of AI and offers a list of specific recommendations 
and action points that should help to establish a Good AI Society (Floridi 
et al. 2018). Al4People proposes five core ethical principles: (1) beneficence 
(promoting well-being, preserving dignity and sustaining the planet); (2) non- 
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maleficence (privacy, security and capability caution); (3) autonomy (the power 
to decide/whether to decide); (4) justice (promoting prosperity and preserving 
solidarity); and (5) explicability (enabling the other principles through intelligi- 
bility and accountability). While the first four overlap with traditional bioethics 
principles, the last one is ‘a new enabling principle for AT (Floridi et al. 2018, 
700). There might be overlap with the Asilomar AI principles, but AI4People 
has come up with a comprehensive list of ethical principles, recommenda- 
tions and action points that can help policymakers to develop and support AI 
projects and initiatives that benefit society. However, they are not without gaps 
and flaws. 


What is Missing in AI Policies and Ethics: Introducing Capitalism 


The overview of AI policies and initiatives aimed at formulating AI ethics, 
helps us understand the debate about what AI we want/need and what it should 
deliver (or what should be avoided). However, something crucial is missing: 
power. This brings us to the crux of the book and the possibilities of critical 
analysis of AI. To bring power into the debate, we must first understand two 
points: (1) the problem of AI ideology and (2) the limitations of ethics. 

Let me start with AI ideology. National policies clearly illustrate that AI is 
seen as an important instrument for positioning countries in terms of what 
type of future society they aim to develop. Here comes the role of ideology. 
While a contested notion, ideology can refer to: ‘worldviews and ideas on the 
one end, to the process of the production of false consciousness on the other 
end of the spectrum’ (Fuchs 2020, 180). In other words, it can have a neutral 
meaning but ideology can also be used to manipulate human consciousness. 
In the latter meaning, ideology is seen as a typical characteristic of capitalism 
and class societies, and it is being used to serve the material interests of the 
ruling class (Fuchs 2020). As discussions of AI often include visions about its 
potential to radically alter societies and economies, we need to be alert to and 
critical towards AI ideology. 

Berman (1992) wrote almost three decades ago that the growing interest 
in AI in capitalist societies can be understood not only in terms of its prac- 
tical achievements but also in the ideological role it plays as a technological 
paradigm for the continuation and reinvention of capitalism. AI as an ideol- 
ogy means that it can be seen as: ‘a potential hegemonic principle within the 
sphere of formal organizations which facilitates the “fit” of human beings into 
the revised structures of a capitalism based on micro-electronic and informa- 
tion technology, and ideologically contains, and significantly mutes, resistance 
and social conflict’ (Berman 1992, 104). The technological paradigm is thus 
a major component of hegemonic ideology that helps to maintain the essen- 
tial structures of the current capitalist system and makes coherent and viable 
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alternatives increasingly difficult to envision. AI ideology thus propagates one 
specific vision of what AI is and what it should do - including serving the inter- 
ests of the ruling class - and discourages alternative visions from materialising. 

Second, we need to be aware of the limitations of AI ethics. Computer ethics, 
the broader field to which AI ethics belongs, is a philosophical field of study 
that deals with the question of ‘how computer technology should be used’ 
(Moor 1985, 266). It investigates social impact but also how policies for ethical 
use of computer technology can be formulated and justified. This is important 
and is why I discuss not only ethical guidelines but also AI policies. AI ethics 
are important as they let us think about what a good society constitutes, how 
we - as members of that society - can live a meaningful and fulfilling life, and 
especially what the role of technology, in general, and AI, in particular, in this 
is (Coeckelbergh 2020). There are, however, problems and limitations with AI 
ethics and how they get linked to policy. 

When it comes to developing AI ethical guidelines, the first question to ask 
is: who is involved? The issue of diversity and inclusion plays out on multiple 
levels. Research by Jobin, Ienca and Vayena (2019). (2019) demonstrates that 
developing AI ethics is concentrated in North America, the European Union, 
Japan and a small handful of other countries. The absence or underrepresenta- 
tion of countries from Africa, Central and South America and Central Asia 
means that large global regions are not invited to contribute to this debate, 
illustrative of a geopolitical power imbalance. There are also questions about 
who exactly is involved in developing the guidelines and whether the panels 
of experts who produce ethical guidelines, are - or are not - representative of 
society. This undermines the plurality that AI ethics aim for. 

Another problem of establishing AI ethics is the speed at which technologies 
are developing (Boddington 2017). Formulating ethical guidelines takes time 
and there is a question of whether or not ethics can keep up with the rapid 
development of technologies. AI policies, just as any policies, face a similar 
challenge and as a consequence they are often reactive rather than proactive. AI 
ethical guidelines also face the problem of ethics washing (Wagner 2018). This 
refers to the practice of exaggerating a company’s interest in promoting benefi- 
cial AI systems (Google’s Advancing AI for Everyone (2020)and Microsofts AI 
for Good (2020) programs, cfr. supra, are often used as examples for this) but 
also when ethics is used as a substitute for regulation, meaning that companies 
highlight how ethically they are acting while simultaneously abandoning their 
legal obligations (for example, not respecting principles in terms of data protec- 
tion). 

The vulnerability of ethics advocates and researchers is illustrated by the case 
of Timnit Gebru. Gebru is well-known for her work on racial bias in technol- 
ogy, such as facial recognition, and has criticised systems that fail to recognise 
black faces. She was fired by Google in December 2020 after sending an inter- 
nal email that accused Google of silencing marginalised voices (Hao 2020). 
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It is clear that we need to be aware of AI ideology and acknowledge that AI 
ethics alone, despite their value and contributions, will not save the world. The 
other problem is about how we move from AI ethics to concrete policies. There 
is no roadmap for what exactly should be done, no precise course of action to 
be taken in policy development (Coeckelbergh 2020). It comes down to who 
has the capacity to influence the actions, beliefs or behaviour of others. Or 
who can influence what type of society we want, and what the role of techno- 
logy such as AI should be in it. Ultimately, this is a question about power and 
who is in control to make decisions. 


We Have to Talk about AI and Power 


The problem of AI ideology and - more broadly - the question of whether 
we need AI and if so, what type of AI we need, illustrates why we need criti- 
cal perspectives on AI. What do I mean by critical? The Frankfurt School has 
been pivotal in the development of critical thinking and theory. According to 
Max Horkheimer (2002), one of the leading figures of the Frankfurt School, 
critical theory distinguishes itself from traditional theory because of its 
focus on human emancipation. The goal of critical theory is to scrutinise and 
understand systems of domination and oppression and to look for ways of 
how to increase liberation and freedom. 

If we make human emancipation central, we need to ask questions about AI 
and power. And this is exactly what is missing in AI policies and ethics: power. 
Power is a contested concept in social theory. In a pragmatic way, Wright (2010, 
111) defines it as: ‘the capacity of actors to accomplish things in the world: 
This is a positive take on power, whereas a lot of definitions of power are nega- 
tive - coercive power, preventing others to act in a certain way (Fuchs 2017). 
In addition to coercive power, Thompson (1995) also talks about economic 
power, political power and symbolic power. Economic power refers to how cer- 
tain individuals and groups in society can accumulate resources for productive 
activity; political power is about the authority to coordinate individuals and 
their interaction; and symbolic power refers to meaning making and influenc- 
ing the actions of others. AI ideology has raised issues of symbolic power, so I 
now turn to economic and political power in the context of AI. 

We need to be aware that AI simultaneously refers to technical approaches, 
social practices and industrial infrastructures (Crawford 2018). The techni- 
cal approaches are straightforward: these are computational systems that use 
data for training machine learning and deep learning algorithms (Alpaydin 
2016). The other two elements need more clarification. The social practices of 
AI refer to the classification systems, developed by humans, which are behind 
the machine/deep learning algorithms and models. Political power asks who is 
involved in developing these classification systems and who decides what they 
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look like (Crawford 2018). Questions about inclusion and representation are 
inherently political questions. AI also refers to industrial infrastructures: the 
infrastructure does not only entail the possibilities of collecting vast amounts 
of data, but also the computational power needed to develop machine/deep 
learning models. Very few companies have simultaneously the computational 
power, access to data and AI expertise (human resources) at their disposal, 
which means that the economic power of these organisations is crucial for the 
development of AI and is highly concentrated (Dyer-Witheford, Kjøsen and 
Steinhoff 2019). 


The Case for a Radical Democratisation of AI 


Asking critical questions about AI with the objective to foster human eman- 
cipation requires us to investigate the political and economic power dynam- 
ics of AI. My point here is that we need to move beyond discussions of what 
beneficial AI means and what opportunities and risks exist in its development. 
We urgently need to think instead about what radical approaches to AI are 
and how we can enable them. Why radical, and what does that mean? Radi- 
cal originates in the Latin word radix, which means root and that is why it 
has been popularised as grasping things at the root. Radical can mean many 
things but here I refer to it as in radical politics (Fenton 2016). Radical politics 
is characterised by its intention of transforming the fundamental principles of a 
political system or a society, often by making use of structural change or radical 
reform - change at the root. 

A radical perspective to AI thus means we need to examine AI through the 
lens of power. Ultimately this comes down to the question of how Al is shift- 
ing power. This is about bringing real change for the better, disrupting power 
dynamics and avoiding an unequal power distribution. We could repeat (and 
slightly revise) William Gibsons (2003) seminal quote ‘AI is already here; it’s 
just not evenly distributed. The question then remains: how can we redistribute 
power in AI? 

My proposal is that if we want to establish AI that transforms society for the 
better and enables human emancipation, we need a radical democratisation of 
AI. This radical democratisation is necessary to avoid power inequalities, in 
other words, to avoid a situation whereby only a few organisations, whether 
governmental or corporate, have the economic and political power to decide 
what type of AI will be developed and what purposes it will serve. 

This is vital in the data and AI sector, which is characterised by a strong 
tendency to establish monopolies. Network effects intensify competition 
between data platforms: the more users on their platform, the more valuable 
they become (Srnicek 2017). More data then also generates more users, which 
allows for the creation of better services. This is called a data-feedback loop. 
Data giants will therefore acquire competitors, which leads to a situation of 
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an oligopoly or even monopoly. This is even more crucial in the AI indus- 
try where few companies have access to data to train machine/deep learning 
algorithms, possess the computing power to deal with massive data sets and 
also to hire the AI talent that is necessary to build AI systems and applications 
(Dyer-Witheford et al. 2019). 

So, what does a radical democratisation of AI actually mean? First, AI and 
the benefits it offers, should be accessible to everyone. Second, AI and the differ- 
ent services that are being developed should also represent everyone. Third and 
last, AI should be beneficial to everyone. These three principles are inspired by 
the late Erik Olin Wright's critique of capitalism. Wright (2019) proposes the 
principles of equality/fairness, democracy/freedom and community/solidar- 
ity as normative foundations for establishing a society that allows its members 
to live a decent life. In the following paragraphs, I briefly unpack these three 
guiding principles. 


#Principle 1: AI Should Be Accessible to Everyone 


This first principle proposes equal access to AI and the benefits it can offer. In 
a decent society, all persons should have broadly equal access to the advan- 
tages and possibilities being created by digital technologies such as AI. This 
means that we need to make sure that all groups in society have access to and 
can use AI. The egalitarian ideal is at the centre of nearly all concepts of social 
justice, including data justice (Taylor 2017), although there are different opin- 
ions about what it means exactly. An important nuance here is to distinguish 
between equal access and opportunity. The former is chosen over the latter as it 
‘is a sociologically more appropriate way of understanding the egalitarian ideal’ 
(Wright 2019, 11). Given the current economic, social and environmental crisis 
we are living in, there should be particular attention to intergenerational and 
environmental justice. The first aspect points to the consequences of techno- 
logical developments for the future generations, whereas the second aspect asks 
for attention for IT and sustainability. This is controversial as AI is both seen 
as a source of and solution for environmental degradation (Dauvergne 2020). 


#Principle 2: AI Should Represent Everyone 


The second principle is centred around democracy and inclusion. In a decent 
society, all members should have a say about what type of AI is being developed 
and what services are being offered. The production and implementation of 
AI must be democratised so that all groups in society are consulted and rep- 
resented, avoiding exclusion. This element of democracy entails two aspects: 
everyone is involved and everyone is represented. The latter aspect highlights 
that when fairness fails, there is a risk of discrimination (Hoffmann 2019). The 
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history of AI is full of examples of how technology is being developed by (pre- 
dominantly) white middle-class men, thereby excluding people of colour and 
minority communities. Wright (2019) also connects democracy and freedom 
in order to reflect the value of self-determination. In this sense, members of 
society should be given the possibility to participate meaningfully in decisions 
that affect their lives. As AI becomes more omnipresent, people should have 
a say about this. Principles such as fairness, accountability and transparency 
(ACM 2020) are key when we want technological development not only to 
represent the people but also guaranteeing control by the people to counterbal- 
ance the power of the state and corporations. 


#Principle 3: AI Should Be Beneficial to Everyone 


The third and last principle states that developments in AI should contribute 
to the well-being of everyone in society. This matches with Wrights (2019) 
ideas of community and solidarity, which are crucial because of their connec- 
tion to human flourishing and of their role in fostering equality and democracy 
(see also principles 1 and 2). Central is the idea that if people cooperate, they 
can achieve more than if they compete, and cooperation also contributes to the 
well-being of all members of society. This means that AI development must be 
organised in such a way that all members of society are able to reap the benefits. 
Another aspect of this principle is the question about how to develop benefi- 
cial machines, in other words, how can we ensure that AI serves the objectives 
of humanity. Stuart Russell (2019, 11) states: ‘machines are beneficial to the 
extent that their actions can be expected to achieve our objectives. According 
to him, this is at the centre of the problem of control in AI and his interpreta- 
tion focuses on the human-machine relationship, as part of being beneficial to 
everyone. Developing AI that is beneficial for everyone, thus includes thinking 
about how to create beneficial machines that serve humanity. 

AI for everyone risks becoming yet another hype, if we let the tech giants 
take over the debate with their slogans such as AI for social good. What they 
are missing is a real vision of democratising technology because they fail to 
understand what AI for everyone really means: putting the human at the centre 
(Pasquale 2020). In one word, this is about power. If we are not talking about 
power, we are not talking about AI for everyone. Critical perspectives require 
us to talk about the human and society. By bringing together diverse critical 
contributions to the debate, this book presents one thing they have in common: 
the idea of putting society first. 


Chapter Overview 


Part 1: AI - Humans vs. Machines consists of four contributions. Andreas 
Kaplan (Chapter 2) goes deeper into the history and definition of AI and 
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elaborates on how humans and machines have to coexist in the age of Al. 
Wolfgang Hofkirchner (Chapter 3) continues the discussion about humans 
versus machines by analysing what Digital Humanism exactly entails. He pro- 
poses dialectical models in order to overcome the human-machine dualism. 
Jenna Ng (Chapter 4) adds to this discussion by elaborating on the rationalisa- 
tion of Aland what this means for creativity. Dan McQuillan (Chapter 5) has 
a different take on humanism and proposes how people’s councils for AI can 
serve solidarity and mutual aid in times of crisis. 

Part 2: Discourses and Myths About AI is comprised of five chapters. Rainer 
Rehak (Chapter 6) stresses the importance but also limitations of metaphors 
when talking about AI and intelligent systems. Angela Daly, S. Kate Devitt and 
Monique Mann (Chapter 7) introduce and discuss their Good Data approach in 
order to overcome the limitations of AI ethics and governance. James Steinhoff 
(Chapter 8) critically analyses the social reconfiguration of AI and discusses 
the central questions about utility and feasibility. Benedetta Brevini (Chapter 9) 
analyses AI policies in Europe and unpacks some of the myths around AI 
that legitimate capitalism. Alkim Almila Akdag Salah (Chapter 10) reflects 
on how the discourses of artistic computational production have changed and 
how myths about AI need to be uncovered in this context. 

Part 3: AI Power and Inequalities involves five contributions. Carrie O'Connell 
and Chad Van de Wiele (Chapter 11) revisit Wiener’s cybernetic prediction 
as the theoretical foundation of AI and make a plea how we need to uncover 
the black box of what is behind prediction and simulation. Jernej A. Prodnik 
(Chapter 12) critically analyses algorithmic logic in digital capitalism, its char- 
acteristics and social consequences. Asvatha Babu and Saif Shahin (Chapter 13) 
investigate biometrics and biopolitics and apply their analysis to a case study 
of the ban on facial recognition in California. Rafael Grohmann and Willian 
Fernandes Araújo (Chapter 14) turn to a discussion of human labour that is 
behind global AI platforms and report about their empirical research on the 
Mechanical Turk in Brazil. Last, Lina Dencik (Chapter 15) also reflects on 
the relationship between labour and AI and proposes the concept of data jus- 
tice unionism to rethink the governance of AI. 
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PART I 


AI - Humans vs. Machines 


CHAPTER 2 


Artificial Intelligence (AI): 
When Humans and Machines 
Might Have to Coexist 


Andreas Kaplan 


Introduction 


Artificial intelligence (AI), defined as a ‘systems ability to correctly interpret 
external data, to learn from such data, and to use those learnings to achieve 
specific goals and tasks through flexible adaptatioy (Kaplan and Haenlein 
2019, 17), will likely have a deep impact on human beings and society at large. 
The recent COVID-19 pandemic has particularly accelerated and accentuated 
society's digitalisation and strongly influences the future relationship between 
human beings and AI-driven machines (Haenlein and Kaplan 2021). 

Various opinions and viewpoints on the future altered by advances in Al exist, 
ranging from horror scenarios as stated by Tesla CEO Elon Musk, to utopian 
scenarios like the vision of Google Chief Engineer Raymond Kurzweil. 
While Musk fears that AI might lead to nothing less than a third world war, 
Kurzweil believes that AI will enhance humans instead of replacing them. 
Expressing these opposing views, in 2018, theoretical physicist Stephen 
Hawking proclaimed that AI can ‘either be the best, or the worst thing, ever to 
happen to humanity’ 
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Clearly, humans will need to coexist with machines. Jobs traditionally done by 
humans will be shifted towards AI systems. Artificial intelligence is already able 
to translate languages, diagnose illnesses, assist in retail (Kaplan 2020c), and 
the like - in several cases, better than the human workforce. Human jobs might 
be created in the future that are unimaginable now, similar to the fact that 
nobody really predicted the job of mobile app designers just a few years ago. 

In this world, AI would rather be augmenting and complementing - rather 
than replacing - humans in their work. In the pessimistic case, i.e., massive 
unemployment, ideas such as universal basic income are already being dis- 
cussed. Fundamental philosophical questions would need to be answered sur- 
rounding life for humans when most of our work is done by AI systems. In 
any case, the State will certainly have to come up with a set of rules govern- 
ing this human < > machine coexistence and interdependence. Society overall 
is thus challenged. 

This chapter has a look at artificial intelligence, its history and its evolutionary 
stages. Furthermore, what challenges might arise in the future when humans 
will have to learn to live among machines and robots will be discussed. This 
will be done by analysing challenges concerning algorithms and organisations, 
challenges with respect to (un)employment, and looking at democracy and 
freedom potentially jeopardised due to AI progress. 


Artificial Intelligence: Definition and Classification 


Artificial intelligence is a rather fuzzy concept, and quite difficult to define. At 
least two reasons can be proposed for the difficulty in formulating a definition 
therefore: firstly, it is not easy to find a clear definition for what intelligence in 
general is, as it depends largely upon the context. Thus intelligence is described 
in several different ways such as the capacity for learning, reasoning, planning, 
understanding, critical thinking, creativity, and last but not least, problem solving. 

Secondly, artificial intelligence is a moving target: advances previously con- 
sidered to AI with time will not be considered as such as soon as we get used to 
them. This phenomenon is known as the AI effect. As McCordick (2004, 204) 
formulated it: ‘It's part of the history of the field of artificial intelligence that 
every time somebody figured out how to make a computer do something - play 
good checkers, solve simple but relatively informal problems - there was cho- 
rus of critics to say, “that’s not thinking” Or as Rodney Brooks, MIT’s Artificial 
Intelligence Laboratory director, explains, ‘Every time we figure out a piece of 
it, it stops being magical; we say, “Oh, that’s just a computation’, and will not 
count as artificial intelligence any longer’ (Kahn 2002). 

One of the prevailing definitions of artificial intelligence, as aforementioned, 
characterises AI as ‘a systems ability to correctly interpret external data, to learn 
from such data and to use those learnings to achieve specific goals and tasks 
through flexible adaptation’ (Kaplan and Haenlein 2019, 17). Several further 
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definitions exist and experts disagree on how to best characterise artificial 
intelligence. By analysing different AI definitions, Russell and Norvig (2016), 
e.g., concluded that there are four main approaches for defining AI, i.e., see it 
as systems that (1) think like humans, (2) act like humans, (3) think rationally 
and (4) act rationally. 

Often terms such as big data, machine learning or the Internet-of-Things 
(IoT) are incorrectly applied as synonyms for artificial intelligence, yet they 
are indeed differing concepts and terms. An Al-driven system needs big data 
from which to learn, which essentially are ‘datasets made up by huge quanti- 
ties (volume) of frequently updated data (velocity) in various formats, such as 
numeric, textual or images/videos (variety)’ (Kaplan and Haenlein 2019, 17). 
Again, a variety of different definitions for big data exists: while one group of 
them focuses on what big data is, a second group stresses what big data actu- 
ally does (Gandomi and Haider 2015). Such big data sets can derive from an 
organisation’s internal databases, third-party data or social media applications 
(Kaplan 2012; Kaplan and Haenlein 2010b). 

Another possibility for obtaining big data is via the Internet-of-Things 
(Krotov 2017; Saarikko, Westergren and Blomquist 2017), which basically is an 
extension of internet connectivity into physical devices and everyday objects 
such as a refrigerator or a heater, equipped with sensors and software to collect 
and exchange data. 

Machine learning, simply put, is ‘methods that help computers learn without 
being explicitly programmed’ (Kaplan and Haenlein 2019, 17), and is applied 
in order to identify underlying patterns within the big data, and as such is an 
essential element of artificial intelligence. A more elaborated definition comes 
from Mitchell (1997, 2) stating ‘A computer program is said to learn from 
experience E with respect to some class of tasks T and performance measure P 
if its performance at tasks in T, as measured by P, improves with experience F? 
AI is much broader than machine learning, as it additionally comprises such 
abilities as the perception of data (e.g., voice/image recognition, natural lan- 
guage processing, etc.) or the control and movement of objects (robotics 
or cybernetics). 

Artificial intelligence can be classified into three types of systems: analyti- 
cal, human-inspired and humanised (Kaplan and Haenlein 2019). Analytical 
AI contains characteristics consistent with cognitive intelligence only: gener- 
ating cognitive representation of the world and using learning based on past 
experience to inform future decisions. Human-inspired AI contains elements 
of cognitive and emotional intelligence: understanding human emotions, 
in addition to cognitive elements, and considering them in their decision- 
making. Humanised AI contains characteristics of all types of competencies 
(i.e., cognitive, emotional and social intelligence), is able to be self-conscious, 
and is self-aware in interactions with others. 

A robot driven by analytical artificial intelligence would be capable of 
answering queries concerning restaurant recommendations based on certain 
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objective characteristics. Human-inspired AI robots could additionally read a 
human’s emotional state via facial recognition or tone of voice, and adapt its 
suggestions, e.g., a human who appears sad or depressed would not enjoy a res- 
taurant with a lively atmosphere, whereas a happy human might totally enjoy 
such an environment. Finally, a humanised robot would understand when it 
was appropriate for it to offer to accompany the human or whenever this would 
not be appreciated, e.g., a couple insanely in love who would rather spend the 
time in intimate togetherness. 

Finally, we must distinguish AI on the lower spectrum from so-called expert 
systems, often wrongly associated with artificial intelligence, as well as on 
the higher spectrum from skills that remain only possible for human beings: 
Expert systems are ‘collections of rules programmed by humans in the form 
of if > then statements’ (Kaplan and Haenlein 2019, p. 18). As these systems 
lack the ability to learn autonomously from external data, they should defi- 
nitely not be counted as AI. Expert systems reconstruct human intelligence in 
a top-down manner (also called the knowledge-based or symbolic approach), 
considering that it can be codified as a set of predefined rules. In contrast, AI 
applies a bottom-up approach (also called the behaviour-based or connection- 
ist approach) and imitates a brain’s set-up (e.g., through neural networks) by 
using large quantities of data to infer knowledge independently. 

The question that arises is what will remain human in the future and what 
cannot be imitated by AI systems, which is quite a tough question to answer. 
Most likely, humans will always have exclusivity when it comes to artistic crea- 
tivity, Albert Einstein having pointed out that ‘creativity is intelligence having 
fun. Currently, it seems very improbable that AI systems will be able to be truly 
creative. But then again, the question is what exactly true creativity is, and who 
will be the judge of it? 


Artificial Intelligence: History and Evolution 


To structure AT’s history, we'll use an analogy of the four seasons: spring, sum- 
mer, autumn and winter (Haenlein and Kaplan 2019). ATs birth period, i.e., 
spring, took place both in fiction as well as non-fiction. Regarding the former, 
Isaac Asimov, an American writer and professor of biochemistry at Boston Uni- 
versity, published ‘Runaround, a story revolving around an Al-driven robot, in 
1942. In this story, Asimov's (1950, 40) three laws of robotics explicitly appear 
for the first time: 


1. ‘A robot may not injure a human being or, through inaction, allow a 
human being to come to harm. 

2. A robot must obey the orders given it by human beings except where such 
orders would conflict with the First Law. 

3. A robot must protect its own existence as long as such protection does not 
conflict with the First or Second Laws? 
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These three laws already hint at the difficulty of humans and robots coexist- 
ing. In any case, the robot in Asimov’s story freezes in a loop of repetitive 
behaviour, as it doesn't find a solution for obeying laws 2 and 3 at the same 
time. Runaround’ is therefore a cornerstone in the history of artificial intel- 
ligence, as it inspired generations of academics and researchers in the domain 
of AI. 

Regarding the real world, we can refer to computer scientist Alan Turing’s 
seminal paper ‘Computing Machinery and Intelligence, published in 1950. 
Therein, Turing describes what now is known as the Turing test, or a test of a 
machine’s ability to exhibit intelligent behaviour equivalent to, or indistinguish- 
able from, that of a human. AI spring’s climax can be pinpointed to the 1956, 
when Marvin Minsky and John McCarthy organised the Dartmouth Summer 
Research Project on Artificial Intelligence (DSRPAI) at Dartmouth College. It 
was at this workshop that the term artificial intelligence was coined. 

After spring, there followed a couple of hot AI summers and very cold AI 
winters. While AI summers were characterised by huge enthusiasm and financ- 
ing of AI, winters were marked by reduced funding and interest in artificial 
intelligence research. The first summer period lasted nearly 20 years. One of 
its successes was certainly ELIZA: Developed in 1966 by German-American 
computer scientist Joseph Weizenbaum, a professor at MIT, this computer pro- 
gram was so good at conversing with a human being that it appeared to pass the 
aforementioned Turing test. 

General hype around AI and its development followed. However, this hype 
was soon replaced by disappointment and disenchantment. AI winter some- 
how had already begun when Marvin Minsky supposedly still contended that 
artificial intelligence could attain a human being’s general average intelligence 
within three to eight years from that moment (Darrach 1970). As we all know, 
this did not occur; AI funding was heavily reduced and another AI summer did 
not happen until the 1980s, when the Japanese government decided to mas- 
sively invest in AI and consequently the US DARPA followed. Success again 
was scarce, and summer was again followed by another cold winter. 

We might have reached ATs autumn, completing the four seasons of arti- 
ficial intelligence (Haenlein and Kaplan 2019), as a result of computational 
strength having constantly increased over recent years, rendering deep learn- 
ing and artificial neural networks possible (Libai et al. 2020). This new era of 
Al is said to have begun in 2015 when AlphaGo, a computer program designed 
by Google, beat a (human) world champion in the Chinese board game Go. 
This event made the news around the world, and regenerated hype around the 
domain of artificial intelligence. 

This hype might continue for quite some time, as we are currently only expe- 
riencing so-called first-generation AI applications, usually referred to as arti- 
ficial narrow intelligence (ANI). Within such systems, AI is only applied to 
very specific tasks such as choosing which news items it will tell an individual 
during his or her morning before-work routine based on the individual's intel- 
lectual preferences. 
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Second-generation AI applications will be able to plan, solve and reason 
problems independently, even for actions for which they have not been pro- 
grammed initially. Such artificial general intelligence (AGI) will thus be able 
to broaden its horizons autonomously, entering new areas and domains. For 
example, an AGI-powered system could, on top of conveying news headlines 
during one’s morning routine, also learn to make coffee for the aforementioned 
individual preparing for work. 

Finally, we might potentially even experience artificial super intelligence 
(ASI), the third generation of AI. Such truly self-conscious and self-aware 
AI systems, outperforming humans in (nearly) all domains, capable of gen- 
eral wisdom, scientific creativity and social skills, could render human beings 
redundant. As such, in our above example, the individual would not need to 
prepare for work anymore, as this could be done entirely by the ASI-powered 
machine or robot (Kaplan and Haenlein 2019). For a detailed discussion on the 
evolution of AI systems, we refer to Huang and Rust (2018). 


Artificial Intelligence: Machines and Humans 


In the future, artificial intelligence will raise several challenges, and humans 
will have to learn to coexist with machines and robots. Pushed by the global 
COVID-19 health crisis, it is clear that AI will deeply impact societies around 
the world (Kaplan 2021). We will discuss some of these questions, looking at 
challenges in terms of algorithms and individual organisations; the employ- 
ment market; and last but not least, democracy and human freedom potentially 
at stake due to advances in AI. 


About Algorithms and Organisations 


When machines and humans coexist, it is important that both do what they are 
good at. As an illustration, let’s have a look at a study by researchers from MIT’s 
Computer Science and Artificial Intelligence Laboratory in cooperation with 
the machine-learning startup PatternEx (Conner-Simons 2016). AI systems 
and humans scored far better in identifying cyber-attacks when collaborating 
than when trying to do so separately. While the AI systems could crawl through 
enormous quantities of big data, humans were better at detecting anomalies, 
playing those back into the system. This iterative and collaborative approach 
was optimal. 

Also, humans are better in behaving ethically and morally, while algorithms 
have problems doing so, as the notion of ethics and morals is difficult to pro- 
gram. Machines, however, are better at, e.g., utilitarian, repetitive tasks. While 
most humans would not consciously discriminate another individual for gen- 
der, sexual orientation, social background, or race, machines, not having a 
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conscience, are more likely to be biased, essentially because the data on which 
they were trained was biased. A study by Wilson, Hoffman and Morgenstern 
(2020) illustrates that several decision-support systems applied by judges 
may be racially biased (as a result of past rulings); and self-driving cars better 
detected lighter skin than darker tones, since their algorithm was trained using 
pictures among which were few people of colour. 

Regulation and guidance is definitely needed in order to avoid such bias, to 
establish a good foundation for machine < > human collaboration. The devel- 
opment of specific requirements with respect to the testing and training of AI is 
likely the preferred approach, as opposed to regulating artificial intelligence 
itself. In addition, we could require AI warranties, consistent with safety test- 
ing in the case of physical goods. Thus, AI regulation could be stable over time 
even if the actual aspects of AI technology change (Kaplan and Haenlein 2020). 


About (Un)Employment 


A tough challenge when human beings coexist with machines might be the 
evolution of the job market. Already, automation in manufacturing has led to 
a significant decrease in blue-collar jobs; advances in AI could lead to a similar 
decrease in white-collar jobs. AI systems already outperform medications in 
the identification of skin cancer and other tasks (Welch 2018). 

For the moment, it appears that the time gain through ATs application is 
used for other tasks within the job, and does not necessarily lead to a human 
beings replacement. The Swedish bank, SEB, e.g., developed AIDA, an AI- 
driven virtual assistant responding to a vast range of customers’ queries, such 
as how to make overseas payments or how to proceed when opening a bank 
account. AIDA is even capable of detecting a customer’s mood by the tone of 
her or his voice and adapting its recommendations and suggestions thereto. In 
around 30% of situations, AIDA is not able to respond or help. In this case, the 
customer is transferred to a human. AIDA’ implementation freed up human 
employees’ time, which they then use for more complex demands, i.e., the 30% 
that exceeded AIDA’s limitations. 

A study by Wilson and Daugherty (2018, 117) suggested that it is in com- 
panies’ interest not to replace employees with AI, as this would not be a long- 
term strategy. Looking at 1,500 corporations, they identified the best improve- 
ments in performance when machines and human beings work together, and 
concluded: “Through such collaborative intelligence, humans and AI actively 
enhance each other’s complementary strengths: the leadership, teamwork, cre- 
ativity, and social skills of the former, and the speed, scalability, and quantita- 
tive capabilities of the latter’ (Wilson and Daugherty 2018). 

However, with advances in artificial intelligence, machines improve, and 
might indeed replace humans in their jobs. It is uncertain that enough new jobs 
at the right skill levels will evolve for everybody, similar to previous shifts in 
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job markets such as the Industrial Revolution. The demanded skill level might 
just be too high for all human beings to be able to find a job not yet done by a 
machine. Or, there just might not be enough jobs left, as more jobs are replaced 
by machines than are newly created. Massive unemployment would result. 

In the short to medium term, regulation could certainly help to avoid mass 
unemployment, at least for a transitional period. Examples are the requirement 
for companies to spend a certain amount of their budgets saved via the help 
of AI on training their workers for higher-skilled jobs; or the restriction of 
the number of hours worked per day in order to distribute the available work 
across the entire population. However, in the longer run, if machines replace 
humans as workers, the idea of a universal basic income will be put back on the 
table. This would trigger a series of fundamental philosophical but also reli- 
gious debates: questions such as the purpose of life, how to feel useful and what 
to strive for, are some issues for which society would have to find answers. Eth- 
ics and education will play an important role in order to tackle these societal 
challenges and questions (Kaplan 2020a). 


About Democracy and Freedom 


Finally, AI progress could represent nothing less than a danger to peace and 
democracy (Kaplan 2020b). There are at least two ways in which artificial 
intelligence might constitute a threat to democracy and its mechanisms, 
endangering the peaceful coexistence of humans and machines: supervision 
and manipulation. 

Using the example of China, we will provide an illustration as to how far the 
possibilities of artificial intelligence reach with respect to control and super- 
vision. AI is largely embraced by the Chinese government, which uses it to 
track and monitor its citizens and inhabitants. For each individual, the Chinese 
government calculates a so-called ‘social credit score’ based on (big) data com- 
ing from various different sources such as health and tax records, social media 
activity, purchasing behaviour, criminal records and so forth. The system also 
uses facial recognition and images of the 200 million surveillance cameras 
mounted across the country for data collection and respective score calcula- 
tion. Good behaviour such as volunteering at an orphanage leads to higher 
scores; bad behaviour such as littering leads to lower scores. In order to fulfil 
the score’s aim, i.e., to encourage good behaviour and citizenship, bad scores 
result in punishments such as not being eligible for bank loans, not being 
allowed to fly or not being hired by public agencies (Marr 2019). 

In addition to control possibilities, artificial intelligence also allows for 
manipulation, as we now constantly experience with the dissemination of fake 
news and disinformation on the various social media platforms (Deighton et 
al. 2011; Kaplan and Haenlein 2010a; Kaplan 2018). Especially in election cam- 
paigns, social media are heavily used to manipulate voters. For example, in the 
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final three months of the 2016 US presidential election, the top 20 false news 
items on only one social medium - Facebook - led to more comments, likes 
and shares than did the 20 most influential news stories from approximately 
20 major actors in the news sector together (including such outlets as the New 
York Times and the Washington Post; Silverman 2016). 

This alone gives enough food for thought regarding the manipulative power 
of AI-based systems. And yet, the next, bigger thing is just around the corner: 
deepfakes, which are AI-based technology used to produce or alter audio or 
video content so that it presents something that did not, in fact, occur’ (Kaplan 
2020b). This technology allows inserting words in audio or even video format 
in an individual’s speech that s/he never actually uttered. Thus, one could make 
a seemingly authentic video of the Pope stating that monogamy is overrated 
and that everybody should have open relationships. What this means for future 
elections and other phenomena is indeed difficult to imagine. 

The above two examples clearly show that artificial intelligence potentially 
leads to issues that do not stop at countries’ borders, with Russia having know- 
ingly been deeply involved in the aforementioned 2016 US presidential election. 
Regulation that applies to some countries only will most likely be ineffective 
in governing the coexistence of humans and machines. Intensive international 
coordination and cooperation in regulation is clearly needed, whenever feasible. 

Such international cooperation might be a challenge. While China and the 
United States are considered as the AI superpowers, they are less known for their 
implementation of AI regulations (Kaplan 2020a). The development of regula- 
tion as well as ethics guidelines falls rather within the expertise of the European 
Union. The EU, however, has far less influence in the actual development and 
elaboration of artificial intelligence. Nevertheless, spill-over effects are possi- 
ble. The EU’s strict General Data Protection Regulation (GDPR), effective since 
May 2018, applies to any corporation that markets products to EU residents, 
regardless of its location. Thus, GDPR influences data protection requirements 
worldwide. As such, the California Consumer Privacy Act (CCPA), which gov- 
erns the most populous US state's data protection since January 2020, is recur- 
rently referred to as Californias GDPR. Government regulation is certainly a 
necessary step. Most likely, whenever society realises the topic’s importance, 
companies will feel obliged to go into the direction of self-regulation, similarly 
to the worldwide impact of citizens’ increased commitment and desire for sus- 
tainability and a stronger protection of the environment. 


Conclusion: Only Time Will Tell 


In this chapter, we introduced the concept of artificial intelligence and how 
it differs from related concepts such as big data, the Internet-of-Things, and 
machine learning. We also surveyed ATs history and evolution before discuss- 
ing the relationship between humans and machines from various angles. 
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Future research will be needed to address the various challenges with regards to 
the development of artificial intelligence. Which formal method can be used 
to test for algorithmic bias? Can we identify simple to use measures to assess 
bias, similar to the way we assess reliability and validity? What is the best way to 
bridge (deep) learning and privacy? Should learning be conducted on the user 
side (with algorithms requiring new data)? Or should data be transferred to a 
trusted intermediary who performs the analysis on behalf of firms? Do users 
need to be compensated in one way or another for data or resources provided? 
Moreover, how can the refusal to share data lead to biases in the data available 
for learning? Which data sources can and should be used for algorithmic learn- 
ing? Are there certain types of data that should be off-limits’? What role will 
interdisciplinary AI teams play in establishing coexistence between humans 
and machines? To mention just a few of the potential future research questions, 
which, in the light of the unprecedented global COVID-19 pandemic and its 
acceleration of society's digitalisation, become of vital importance. 

At least for the moment, it looks as if AI-driven machines will enhance human 
work instead of replacing it. This is also the opinion of John Kelly, vice presi- 
dent of IBM, who stated, ‘Man and machine working together always beat or 
make a better decision than a man or a machine independently’ (Waytz 2019). 
Moreover, according to a recent Accenture study, more than 60% of employees 
believe that AI will have a beneficial impact on their work and jobs (Shook and 
Knickrehm 2017). 

COVID-19 impressively showed that artificial intelligence has played an 
important role in tackling this unprecedented health crisis on a global level. As 
such, researchers worldwide made use of AI to efficiently identify potentially 
infected humans, analyse the virus, test possible treatments and therapies, and 
more generally to find strategies to fight the pandemic. AJ Venkatakrishnan, 
e.g., applying AI, discovered that a mutation of the original virus would mimic 
a protein which the human body uses to regulate its fluid and salt equilibrium 
(Cha 2020). However, the application of artificial intelligence also showed its 
connected impact on individuals’ daily lives as well as on such questions as 
data security and privacy. Regulation for the human-machine entanglement 
is clearly needed. 

Furthermore, an example at Mercedes-Benz clearly shows that the replace- 
ment of the human workforce is still not as easy as sometimes claimed, 
and that indeed, currently, human < > machine coexistence is here. Nor- 
mally, in the automobile manufacturing process, robots and automation are 
common. However, Mercedes-Benz key accounts increasingly demand more 
customisation - which the robots were not able to deliver. 

Therefore, the German automobile giant decided to replace the fully auto- 
mated process with ‘cobots, or collaborative robots, which are robots designed 
to physically interact with human beings in a shared workspace. These cobots 
are controlled by humans, and are to be considered an extension of the 
human's body, facilitating the carrying and moving of heavy car parts. This 
form of human < > machine collaboration enables an efficient and productive 
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customization process, responding in real time to customers’ precise choices 
with regard to leather seats, tyre caps, and so forth. 

As in the automotive sector, AI will certainly trigger changes and evolutions 
in the upcoming years in many sectors. Without a crystal ball, it will be difficult 
to know where and how the coexistence of humans and machines will evolve. 
However, it is crystal clear that the business world (and society at large) will 
need to constantly adapt to advances in Al in order to keep up with the pace 
(Kaplan and Haenlein 2020), or, to quote Benjamin Franklin: ‘When you're 
finished changing, you're finished. 
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CHAPTER 3 


Digital Humanism: Epistemological, 
Ontological and Praxiological 
Foundations 


Wolfgang Hofkirchner 


Introduction 


It seems a common agreement that due to certain progress made in Artificial 
Intelligence (AI) and related fields mankind is facing a blurring of the human 
and the machine such that humanism is put under pressure. Is humanism out- 
dated and can it be renounced? Or does it only need an update? And if so, an 
update in which direction? 

There is discussion abound with pros and cons concerning technological, 
military, sociological and philosophical aspects of AI, Trans- and Post Human- 
ism (Hofkirchner and Kreowski 2020). And there is a candidate for updating 
humanism - Digital Humanism. 

This term popped up in a Gartner Special Report published in April 2015. 
The report had the title ‘Digital Business: Digital Humanism Makes People Bet- 
ter, Not Technology Better and its summary makes clear what Digital Human- 
ism was supposed to be about and what it is was not supposed to be about: 
‘Digital humanism is the recognition that digital business revolves around 
people, not technology. CIOs and business leaders who recognise that digital 
business revolves around people's value will see employee capabilities translate 
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into product, service and market gains? The term did not refer to humanism as 
a philosophical tradition. 

This is in stark contrast to the intentions of German philosopher and former 
minister Julian Nida-Rümelin who had used the term for a long time in lec- 
tures before he published, together with Nathalie Weidenfeld, a book with the 
title ‘Digitaler Humanismus (2018), for which the authors received the Bruno 
Kreisky Prize from the Karl-Renner-Institut, Wien. The German term inspired 
Hannes Werthner, the then Dean of the Faculty of Informatics at the Vienna 
University of Technology (TU Wien), to translate it into English when he con- 
vened a workshop in April 2019 that ended with a manifesto - the Vienna Man- 
ifesto on Digital Humanism. 


This manifesto is a call to deliberate and to act on current and future 
technological development. We encourage our academic communities, 
as well as industrial leaders, politicians, policy makers, and professional 
societies allaround the globe, to actively participate in policy formation. 
Our demands are the result ofan emerging process that unites scientists 
and practitioners across fields and topics, brought together by concerns and 
hopes for the future. We are aware of our joint responsibility for the cur- 
rent situation and the future - both as professionals and citizens. 


We must shape technologies in accordance with human values and 
needs, instead of allowing technologies to shape humans. Our task is 
not only to rein in the downsides of information and communication 
technologies, but to encourage human-centered innovation. We call for 
a Digital Humanism that describes, analyzes, and, most importantly, 
influences the complex interplay of technology and humankind, for a 
better society and life, fully respecting universal human rights. 


Given these quotations from the manifesto (Vienna Manifesto on Digital 
Humanism n.d.), Digital Humanism, meaning an update of humanism - of the 
image of man - in the age of digitalisation, promises to become a label for an 
answer to the questions raised above in a direction worth supporting, a direc- 
tion not technology-driven but aiming at promoting a humane digitalisation. 

This chapter at hand intends to contribute to philosophical, in particular, 
philosophy of science aspects such as praxio-onto-epistemology developed 
from the author elsewhere (Hofkirchner 2013), as sound foundations for 
such an updated humanism. It aims at clarifying the following problem: 

How can a relation between human and machine be established in thinking 
and acting such that fallacies in theorising are avoided? 

There are three ways of framing, modelling and designing the human and 
the machine, in particular, computer, cyber technology, digitalisation, in rela- 
tion. One way is conflation - the false assertion of identity of what is different. 
Another way is the disconnection - the false assertion of a difference of what 
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Table 3.1: Frames, models and designs in the perspective of conflations, 
disconnections and combinations. 


Conflations Disconnections Combination 
Anthropo- Techno- Anthropo- | Techno- | Man-machine- | TechnoSocial 
morphism morphism | centrism centrism | hybridity Systemism 
Cross-disciplinary Mono- and multi-/inter-disciplinary | Transdisci- 
plinary 
Frames 
Sociological | Technologi- | Sociologism | Techno- | Methods mix | Systemic 
colonisation | cal takeover logism complements 
Monistic Dualistic Dialectical 
Models | Anima Mechanism | Pride of Post- Man-machine- | Systems of 
creation human _| hybrids systems 
Assimilative Segregative Integrative 
Design | techno sapiens | homo deus | Supremacy | Singulari- | Man-machine- | TA and design 
tarianism | hybridisation | loop 


is identical. And the last but not least way is the combination - the exercise to 
find out what is identical (what do both sides have in common though they 
might differ in some respects) and what is different (though they might have 
something in common). This is the only way with the prospect of transgress- 
ing falsehood. 

The next three sections discuss these three ways in more detail. Frames, mod- 
els and designs are dealt with. They refer to epistemological, ontological and 
praxiological issues respectively, (see Table 3.1). 


Conflations 


It is conflation if what is widely known as anthropomorphism is the case - the 
assertion of a human property in a realm where it is not an essential property. 
But there is also a second kind of conflation - the assertion of a machine prop- 
erty ina realm where it is not an essential property, which might, in analogy to 
the term anthropomorphism, be labelled technomorphism. Both kinds of con- 
flation should not be conflated. They belong to different ways of thinking and 
acting and yield different results. Anthropomorphism is based upon a projec- 
tion, while technomorphism is based upon a reduction. A projection projects 
higher complexity onto lower complexity so as to simulate higher complexity, 
while a reduction reduces higher complexity to lower complexity so as to simu- 
late lower complexity. In the first case, you have an upgrading of complexity, 
whereas, in the second case, you have a downgrading. 

Let’s now turn to the discussion of how the anthropomorphic and techno- 
morphic conflations work when framing, modelling and designing the relation 
of human and machine, one by one. 
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Cross-disciplinary Frames 


Both anthropomorphism and technomorphism claim to use a common episte- 
mology, a general frame of investigation for both human and machine. 

But in the case of anthropomorphism, that frame is different from the tech- 
nomorphic frame. Anthropomorphism extends the frame normally used in 
social sciences and humanities to information technology. It does so on the 
underlying assumption that those frames that are apt for social phenomena are 
also apt to investigate phenomena that are technical. That is, it looks upon tech- 
nical phenomena as if they were social ones and in doing so it carries over to 
them expectations that they would show what social phenomena are showing. 
Thus, anthropomorphism is open to apply the term intelligence when speak- 
ing of artificial phenomena that shall be compared with human intelligence. 
Attempts to establish electronic personhoods for AI applications are examples 
of our inclination to anthropomorphising. 

In the case of technomorphism, the situation is reversed. Methodologies that 
are usually built for technological research cover social phenomena. Thus, they 
convey expectations of technicality when applied in inquiries into social phe- 
nomena. Social phenomena are deemed engineerable. Human intelligence can 
be researched as if a phenomenon of an artefact. The human brain project of 
the EU pertains to this kind of fallacy. 

In any case, the respective frame cuts across social as well as technologi- 
cal phenomena. The different disciplines of science are conflated - either 
to a social science take of technical phenomena or a technological take of 
social phenomena. 

The current dominant approaches in social, human and arts research, on the 
one hand, and in natural science and technology, on the other hand, are still 
suffering from the divide between the two cultures as batptised by C. P. Snow 
(1998) in the last century. The first culture has been laying the emphasis on a 
qualitative methodology, while the second culture has been fixing a quantita- 
tive methodology as a must. Of course, there have been transgressions of the 
boundaries; ecology, pharmaceutics, or parts of physics have partly become 
friends with anthropomorphisations - one step towards esotericism; psychol- 
ogy, economics, or empirical social research are accustomed to performing as if 
belonging to natural sciences - one step forward to their computerisation and 
technisation as might be the case of computational social science. 

Though the intent to find a general methodology for research in humans 
and machines is commendable, neither attempt to let methodology stretch 
across its own boundaries is a solution, as long as they are not taken up with a 
third culture. 

By applying a method of generating knowledge you will not get findings 
other than those that are due to the method applied. The method applied is the 
necessary condition on which a particular model is based. 
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Monistic Models 


Both anthropomorphism and technomorphism come up with a monistic ontol- 
ogy. Being a human and being a machine are assumed to be identical. However, 
the identity is constituted on the basis of their different framing ways. 

Anthropomorphism is prone to stating that any machine resembles essen- 
tially a human. Technomorphism is in favour of saying that any human resem- 
bles essentially a machine. Anthropomorphism projects essential human 
features - like disposing of intelligence - onto machines. Technomorphism 
reduces essential human features - like disposing of intelligence - to features 
of machines. 

Projection and reduction follow a stepwise order of mediation. 

The anthropomorphic projection runs through the following steps: 


e In a first step, the essential features of sociality of humans, namely, that they 
live in society governed by social relations, are projected onto the individual 
actor, thereby making her a social being. 

e In a next step, the essential features of this individual actor as social being 
are projected onto the human body of the individual as a living being, by 
which she is viewed as a bio-social being. 

e In a further step, the essential features of this bio-social being are projected 
onto the physical substrate of the bio-social being so as to yield a physico- 
bio-social being. 

e In a final step, the essential features of this physico-bio-social being are pro- 
jected onto any mechanistic compartment of the physico-bio-social being, 
so as to blur the distinction between the human and the machine. 


Human(like)ness is conferred from human intelligence via mechanisms that 
work in the human body and might be part of human intelligence to the 
mechanics of artefacts. So, AI can be imagined as being humanly animated. 
Anthropomorphism is hence close to ideas that conceive our planet as a living 
organism, or the universe as ensouled or as a big natural computer. 

The technomorphic reduction is carried out by a concatenation of the fol- 
lowing steps: 


e First, the essential features of the society of humans are reduced to those of 
the individual actor. This is an individualistic fallacy. 

e Second, the essential features of the individual social actor are reduced to 
those of the human body. This is a fallacy of biologism, since the social fea- 
tures of the individual are narrowed down to biotic features. 

e Third, the essential features of the human body are reduced to those of its 
physical substrate. This is a fallacy of physicalism, since the biotic features 
of the body are narrowed down to physical features. 
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e Fourth, the essential features of the physical substrate are reduced to those 
of mechanisms. This is a fallacy of mechanicism, since the physical fea- 
tures of the substrate are narrowed down to mechanical features. The term 
mechanical denotes here having the property of strict determinism. The 
physical world is not full of mechanisms only. 


According to technomorphism, human intelligence boils down to a mere 
mechanical capacity that artefacts can be made capable of. 

Monistic models that conflate human and machine form necessary condi- 
tions for particular design practices. 


Assimilative Designs 


Both anthropomorphism and technomorphism recommend an indiscrimina- 
tive strategy when it comes to praxiology. Praxiology is a term that comprises 
those parts of philosophy that, apart from epistemology and ontology, deal 
with issues that are suitable for the general guidance of human practice such 
as values and norms; ethics, aesthetics or axiology belong to this class of philo- 
sophical disciplines. Praxeology is the name of a certain school of praxiology. 

According to the conflationist suggestions, human and machine shall be 
treated in one and the same way. But they have different beliefs of how the 
activity shall be guided. 

Anthropomorphism renders the humans colonised by machines, if it declares, 
in account with its projective ontology and epistemology, that machines shall 
be treated like humans. By adding to machines a value that is improper, humans 
become assimilated to them. The design of machines aims at producing ‘techno 
sapiens’ (Wagner 2016) - autonomous beings endowed with AI that delimits 
the generic autonomy of humans and ignores the fact that the evidence of intel- 
ligence that is based on the observation of behaviour only is no robust evidence 
at all (think of the Turing test that, actually, proves how easily human compre- 
hension can be fooled). 

The technomorphic credo runs the other way around: not machines shall 
be treated like humans but humans shall be treated like machines. This is 
at the same time the motto of transhumanism. The design aims at ‘homo 
deus’ (Harari 2016) by perfecting the species with artificial means, includ- 
ing the enhancement of their intelligence. Humans shall be engineered to be 
optimised. In that humans shall become machines themselves, humans are 
assimilated to machines, again. 


Disconnections 


Disconnections are the opposites of conflations. They come up as results of dis- 
junctive ways of thinking and acting. The human and the machine are disjoined 
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and separated so much that they don’t seem to have anything in common. Dis- 
connections come in three variants - one comes as focus on the human with 
disregard for the machine, another as focus on the machine with disregard for 
the human, and a last one as focus on an interaction of disjoint humans and 
machines. The first disconnection is anthropocentric, the second technocen- 
tric, and the third hybrid, that is, human-machine-interactive. As to complex- 
ity, all variants presume self-contained degrees of complexity independent of 
any other complexity. 
Let's again discuss the frames, models and designs of the three variants. 


Disciplinary Frames 


In epistemology, all variants agree that data of the human or data of the 
machine need each a frame of their own. In contrast to the cross-disciplinarity 
of the conflationist frames, they represent different supporters of discipli- 
narity. Anthropo- and technocentrism form a group of adherents of mono- 
disciplinarity and hybrid human-machine-interactivism follows multi- or 
inter-disciplinarity. 

Mono-disciplinarity means intra-disciplinary research, it goes inside one 
discipline. Anthropocentrism claims social science and humanities methods 
for social and human data, technocentrism claims technological methods for 
technical data. Since in the first case the role of the lead science is attributed 
in that context often to sociology, the anthropocentric frame can thus 
run under the label sociologism. The technocentric frame might be called - 
analogically - technologism. Sociologism gives technological issues no atten- 
tion. Thus, it does not care about artificial intelligence. Technologism is another 
methodological choice that is found at departments of computer science and 
others throughout the world. It is nourished by the condition of competitive 
excellence in one’s own discipline and AI is one of the important fields and it 
has been diversifying into related fields like Autonomous Systems, Deep Learn- 
ing etc. Both sociologism and technologism add to the existence of two cultures 
instead of trying to overcome them. 

Multi-disciplinarity ‘includes several separate disciplines, e.g., when 
researchers from different disciplines work together on a common problem, 
but from their own disciplinary perspectives (Burgin and Hofkirchner 2017, 
2). Multi-disciplinarity is a rather undeveloped state of working together. 
Inter-disciplinarity ‘involves interaction and coordination between several 
disciplines aimed at the development of knowledge in these disciplines, e.g., 
when researchers collaborate transferring knowledge from one discipline to 
another and/or transforming knowledge of one discipline under the influence 
of another discipline’ (Burgin and Hofkirchner 2017, 3). But despite cursory 
exchanges at points of intersection, disciplines keep themselves reciprocally 
exclusive without significant change - think of Science-Technology-Society, of 
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Informatik und Gesellschaft in German-speaking countries and else. Hybrid 
human-machine-interactivism tries a mix of particular frames. As long as a 
third culture will not be under consideration, a mixed frame will not transform 
the encounter of human intelligence and Al into a consistent approach. 

Those deficient epistemological frames are a shaky premise for ontologies. 


Dualistic Models 


As to ontologies, anthropo-, technocentric and interactivist models are used to 
dualism instead of monism as in the case of anthropo- and technomorphism. 
Human and machine are assumed to be disjunct and to belong to different 
classes of the real world. 

The main point of anthropocentrism is that the human is incommensurable 
with a machine. Humans and society are modelled as something completely 
different from a machine. Man is not a machine. Man is unique. Idealistic and 
spiritualistic positions would share such an approach. Humans are regarded as 
sentient, robots as corpses. Human intelligence is not mechanical. 

What the anthropocentric ontology holds for the human, technocen- 
trism holds for the machine. The machine is modelled as something that 
avoids human error. This makes machines unique. Technophilia as in trans- 
and posthumanism are examples of such a position. Machine intelligence is 
not human. 

While the anthropocentric and the technocentric models hypostatise the 
uniqueness of either the human and social or the machine, the hybrid, interac- 
tivist model focuses on the interaction of both sides that enter the interaction 
asindependent entities. But since the different degrees of complexity of both sides 
are not taken into consideration, a plural network is hypostatised that obscures 
the effective working of the interaction. This is the result of using the frames of 
multi- and inter-disciplinarity. Examples are the flat ontologies in Bruno 
Latour’s Actor-Network Theory (ANT) (Latour 2006), which conceives humans 
and machines as ‘actants, as well as sociomaterialism (Barad 2012, Suchman 
2007), which conceives of generic ‘intra-actio of agents with their ecologies. 

Dualistic models that cannot avoid the disconnection of human and machine 
are the proper basis for designs that segregate. 


Segregative Designs 


Anthropocentric, technocentric and interactivistic designs follow the 
pattern of segregation. The human and the machine shall be treated in discrimi- 
native ways. 

Anthropocentrism holds that the human shall be treated better than the 
machine. Man is the pride of creation, as theocratic beliefs have been formulating. 
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The human shall be perfected without resorting to technology. Social pro- 
cesses are placed over and against technological ones - technology is treated 
as trumpery, engineering might even be dangerous. AI is not needed or might 
devaluate the position of human intelligence. 

The technocentric position is the opposite of anthropocentrism: The 
machine shall be treated better than the human. The machine is to be per- 
fected to be devoid of human error. If a machine is liable to failure, then it is 
because of errors of the operators, that is, humans, because of programming 
errors that are the fault of humans, or because of material defects that are, in 
the end, due to faults of humans, again. Machines can, in principle, and they 
do so in reality, outperform humans. Intelligence of machines will render the 
intelligence of humans obsolescent. That is the credo of posthumanism and 
singularitarianism - a kind of Nietzsche's Ubermensch but ex machina, that 
is, from the machines, robots, autonomous systems, AI. 

The interactivistic position does not prioritise either side: The human and the 
machine shall be treated on an equal footing. However, doing so falls back into 
conflationist positions as to the interplay of social and technological practices. 
Anyway, in hybrid networks, design levels up machines or levels down humans. 
According to the famous saying of Latour that it is not me who shoots with the 
pistol but it is the pistol which (maybe better: who?) shoots with me, it is not 
humans who make decisions but intelligent devices whose decisions we just 
adapt to or execute (e.g., in the case of so-called expert systems in health care). 


Combination 


In contradistinction to conflations that frame, model and design human and 
machine on the sole basis of supposed identity of their degrees of complexity as 
well as in contradistinction to disconnections that do the same on the sole basis 
of a supposed difference of their degrees of complexity, a third way of thinking 
and acting orients towards the acceptance of identity and difference of their 
degrees of complexity at the same time - an enterprise of integration of human 
and machine. Integration is a combination that does justice to both what is 
universal and what is particular to human and machine. 

The term that is chosen here to characterise the combinations with regard 
to the epistemological, ontological and praxiological aspects is techno- 
social systemism. 


Transdisciplinary Frames 


Techno-social systemism transgresses cross-disciplinarity and disciplinar- 
ity, in particular, it needs more than multi- or inter-disciplinarity - it needs 
trans-disciplinarity. Transdiciplinarity ‘encompasses problems from different 
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disciplines but goes on a higher level than each of these discipline goes. In 
other words, transdisciplinarity treats problems that are at once between the 
disciplines, across the different disciplines, and beyond any of the indi- 
vidual disciplines involved. It is aimed at understanding of broad spheres of 
the world directed at the unity of knowledge’ (Burgin and Hofkirchner 2017, 3). 

A transdisciplinary frame needs systemism in the methods, that is, the 
assumption that different disciplines are to be interrelated in a systemic 
framework that provides what they have in common and grants, at the same 
time, relative autonomy to each discipline according to their place in the over- 
all framework. Both social science and technology need to complement each 
other in order to constitute the big picture. Social data, technical data and data 
of the techno-social interaction are needed in unison. 

Systemism has the potential to combine those data by combining the discipli- 
nary approaches in question. It gives the whole edifice of sciences a new shape, 
from philosophy over the formal, real-world and applied sciences further on 
to disciplines on sub- and sub-sub-levels. It turns the formal sciences into a 
systems methodology, the real-world sciences into systems sciences and the 
applied sciences into sciences of artificial design of those systems. In such a way, 
the foundation of a science of techno-social systems is laid. Social science and 
engineering construe a common understanding of the systemic relationship of 
society and technology such that social systems science informs ‘engineering 
systems science by providing facts about social functions in the social system 
that might be supported with technological means’; engineering systems sci- 
ence provides ‘technological options that fit the social functions in the envis- 
aged techno-social system’; and social systems science investigates, in turn, ‘the 
social impact of the applied technological option in the techno-social system 
and provide[s] facts about the working of technology’ (Hofkirchner 2017, 7). 
Hence, the epistemology of techno-social systems research paves the way for an 
ontology of human and machine, and for a praxiology of an integrated cycle of 
technology assessment and technology design. 

Thus, techno-social systemism claims for a single frame for social and tech- 
nical data that are comprised on a systemic meta-level. 

The way is open to an unfettered scientific understanding of human intel- 
ligence, artificial intelligence and their relationship. 


Dialectic Models 


A techno-social systems ontology cannot resort to monism nor to dualism. It 
requires dialectic. A dialectical relationship goes beyond duality in that sides or 
parts are not completely separate. And neither are they brought together by 
operations on the surface. They hang together intrinsically, but asymmetrically, 
over steps of emergence. They are evolutionary products, they give rise to evo- 
lutionary products, they are nested one in another in line with their complexity. 
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Techno-social systems are social systems. They emerge from social systems 
when technologies of any kind are inserted into the social systems so as to 
improve the functioning of the social systems to reach a certain goal through 
the mediation of these technologies. These technologies transform those very 
systems into techno-social ones. These technologies are devised and devel- 
oped to functionalise a certain cause-effect-relationship of the real world 
as artificial mechanisms in which the effect becomes the goal and the cause 
becomes the leverage. In order to serve effectively and efficiently the attain- 
ment of the desired or needed goal, artificial mechanisms are prepared to func- 
tion as strictly deterministic as possible. In this respect, artificial mechanisms 
resemble natural mechanisms - the latter work according to strict determinism 
too. An artificially prepared mechanism is what is usually known under the 
term machine. 

Thus, techno-social systems integrate humans and machines. Humans are 
products of evolution, machines are products of humans. Techno-social sys- 
tems integrate them in line with their ontic features according to their evolu- 
tionary history. Humans and machines share, or have distinct, physical, biotic 
and social features (Hofkirchner 2020). 

Let’s first discuss their physical features: 

Humans and machines share the fact that they are entities and embrace pro- 
cesses that belong to the physical realm. However, they differ essentially with 
regards to the specifics of their being physical and behaving physically. Making 
use of a distinction of Rafael Capurro (2012), humans and society can be inter- 
preted as an agens - that is something that displays agency by itself - whereas 
a machine can be interpreted as a patiens - that is something that does not 
display agency and is passive. This is indicated by the following: 


e Humans and society are able to organise themselves, that is, to build up 
order by using free energy and dissipating used-up energy, whereas 
machines cannot self-organise. 

e Humans and society are made up of elements that produce organisational 
relations that constrain and enable synergy effects and they can constitute 
superordinate systemic entities, whereas machines are made up of modules 
that are connected in a mechanical way. 

e Humans and society function on the basis of less-than-strict determi- 
nacy, which yields emergence and contingency, whereas machines are 
strictly deterministic and cannot behave in an emergent or contingent 
manner. 


Second, let’s turn to the discussion of biotic features: 

Humans and society are physical entities and activate processes that belong 
to the biotic realm. Machines may, but do not need to, have parts that belong to 
the biotic realm. Even in cases where they do so, they differ essentially in qual- 
ity. Humans and society are agents that are autonomous in the true sense of the 
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word (Collier n.d.), whereas machines are heteronomous mechanisms that can 
thus not show any degree of autonomy, as follows: 


e As with any living system, humans and society are able to maintain their 
organisational relations by the active provision of free energy, whereas 
machines cannot maintain themselves. 

« As any living system, humans and society are able to make choices accord- 
ing to their embodiment, their embedding in a natural environment and 
the network of conspecifics, whereas machines cannot choose. 

e As any living system, humans and society are able to control other systems 
by catching up with the complexity of the challenges they are faced with by 
the other systems, whereas machines cannot catch up with complexity and 
are under control by organisms. 


And, third, let's discuss one last category of features - the social one: 

Humans and society are not only physical and biotic, they are the only physi- 
cal and biotic systems on Earth that belong to a specific, the social realm, too. 
They are, essentially, social agents, that is, actors. Machines are social products, 
artefacts, that are made by actors, but they do not possess the agency of actors. 
This is implied by the following: 


e Humans in society constitute — by action, interaction and co-action with 
other actors - social agency that reproduces and transforms the structure 
of the social system (social relations), that, in turn, enables and constrains 
the social agency, whereas machines do not partake in the constitution of 
society but support the action, interaction and co-action of actors. 

e Humans in society provide the commons as effects of social synergy, 
whereas machines support the provision of commons and pertain them- 
selves to the commons. 

e Humans in society are the driving force of social evolution, including the 
evolution of culture, polity, economy, ecology and technology, whereas 
machines are driven by social evolution. However, they can even play a sup- 
portive role in changing the quality of the social system. 

e Humans in society reflect upon the social structure, whereas machines do 
not deliberate but support the thought functions of actors. 

e Humans in society set off the transition into actuality of a societal option 
of choice out of the field of possibilities, whereas machines do not directly 
trigger emergence. 


As to the role of AI in the context of techno-social systems, we can conclude 
that artificial intelligence is and will be a mediation of the collective intelligence 
actors are capable of but is not and will never be (a property of) an actor itself. 
What is labelled AI, is nothing that can become independent and achieve a life 
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of its own. However, it promotes the intelligence of the social system. In this 
vein, Francis Heylighen (2015, 2016) rejects the idea of a singularity by which 
a single supra-human artificial intelligence seems purportedly possible, since 
intelligence is and will be distributed over social actors that cyber technology 
merely connects, which means that the emergence of a ‘global brain’ remains 
rooted in humans. From this dialectical point of view, what is in statu nascendi 
is a social suprasystem that would be global, notwithstanding the technological 
infrastructure of a global brain. 

Dialectic models are the proper contributions to a paradigm shift towards 
the third culture. 


Integrative Designs 


Techno-social systemism demands an integrative way of thinking and acting. 
It demands responsibility in two different respects: first, the responsibility for 
the functionality of what shall be designed - does the mechanism effectively 
and efficiently serve the purpose for which it shall be designed? This is a mat- 
ter of fact. However, since the question of how functional technology is can be 
answered in a decontextualised manner from a mere technical point of view, 
a second respect is required: The responsibility for the meaningfulness, for the 
social usefulness of what shall be designed - does the purpose for which tech- 
nology shall be designed also make sense, that is, does it promote a social value, 
does it conform with a social norm? The whole picture of praxiology can be 
seen only when in the context of the social. The default value of meaningful 
technology is to serve the vision of a good society, of individuals living a good 
life and of cultivating the common good. Such an alter-humanism instead of an 
old-fashioned humanism or post-humanism is compatible with the third cul- 
ture - alter-humanism harnesses tools for conviviality (Illich 1973). This means 
that techno-social systems integrate humans and machines according to their 
appropriate treating. The check of that necessitates an integrative technology 
assessment and technology design. 


Conclusion 


A review of possible ways to establish a relation between human and machine 
clarifies the shortcomings, if not the stubbornness of old-fashioned humanism, 
on the one hand, and anti-humanism in a modern disguise, on the other, when 
an identity of human and machine is affirmed at the cost of their difference that 
is negated - done so by conflations - and when the difference between human 
and machine is affirmed at the cost of their unity that is negated - done so by 
disconnections. The way out is the establishment of a relation through affirm- 
ing both the identity of, and the difference between, the two sides - as done by 
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combinations. Combinations provide the proper basis for a humanism that is 
up to the challenges of digitalisation - Digital Humanism. 
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CHAPTER 4 


An Alternative Rationalisation of 
Creative AI by De-Familiarising 
Creativity: Towards an Intelligibility 
of Its Own Terms 


Jenna Ng 


“There, look! we could say. ‘Look at this art! How dare you claim these children are 
anything less than fully human?’ 
- Kazuo Ishiguro (2005, 238) 


Introduction 


This chapter formulates an alternative understanding of creative Artificial Intel- 
ligence (AI) by examining how the computational terms of AI may be rational- 
ised in a framework intelligible to humans. The level of algorithmic processing 
today presents two tensions which hinder a full comprehension of creative AI. 
The first is the still formidable lack of transparency of ATs workings, as noted 
by many scholars å la the algorithmic ‘black box’ (Pasquale 2015; Diakopoulos 
2016; Brill 2015; Ananny and Crawford 2018). The second is the increasing 
lack of human intervention in the algorithm’s processing not only through the 
seemingly unfathomable operation of its ‘black box, but also through algo- 
rithms learning from other algorithms, such as by way of a Generative Adver- 
sarial Network (GAN). The result is to consider anew how computers may be 
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considered to be autonomous creators - to be genuinely creative in and of itself, 
or creators ‘in their own right’ (Veale and Cardoso 2019, 2). Or, as Hofstadter 
(2000) writes: ‘It [the mechanical substrate of creativity] may not constitute 
creativity, but when programs cease to be transparent to their creators, then the 
approach to creativity has begun’ (669; emphasis added). 

Recent innovations in creative AI bear out both tensions, where the algo- 
rithm generates creative decisions, say on note, word or paint placement, 
out of its own processing of the dataset it receives, and in ways not entirely 
understandable to humans. This level of processing may be contrasted with 
how computers had previously produced creative work, or in what is known as 
automated creativity. As early as the 1950s, computers have produced creative 
outputs, such as music, by running codes of basic material and stylistic param- 
eters which enabled the generating of ‘raw materials. These musical ‘materi- 
als’ were then modified and assembled by human composers into recognisable 
pieces of music (Alpern 1995). In these cases, the computer was specifically pro- 
grammed to produce the creative output, following instructions on where and 
how to place notes or dabs of paint, even if those instructions may be rules of 
randomness.' The research field of Computational Creativity recognises such 
programming as ‘pastiche, where the computer’s creativity is a ‘mere appear- 
ance; and only due to some specifiable slice of the programmer’s own creativity 
having been imprinted onto the algorithmic workings of the system. (Veale and 
Cardoso 2019, 4). 

Conversely, current creative AI operates as neural networks which discern 
specific patterns out of processing large datasets of relevant outputs, thus 
‘learning’ across complex nodal networks of ‘reward’ and ‘punishment’ the 
placements of note, paint and words for producing the creative output in keep- 
ing with the patterns they have ‘learned’ A couple of examples to illustrate this: 
in 2016, the Project Magenta team at Google unveiled a 90-second melody pro- 
duced by a computer to which they fed ‘some 4,500 popular tunes’ and ‘seeded’ 
with four musical notes. By processing the large database of tunes, the network 
‘composed’ its melody by discerning patterns of musical rules and constraints 
in ways never specifically programmed into it. The algorithm may also learn from 
other algorithms, such as the algorithmically-produced portrait of Edmond 
Belamy in 2018. In this case, a discriminator algorithm was first trained on a 
database of 15,000 portraits (painted by human artists) that had been uploaded 
to it (by human computer scientists). The discriminator algorithm was then 
used to ‘train’ a separate generator algorithm which ‘learnt’ paint placements 
and so on based on ‘reward’ and ‘punishment’ feedback from the discriminator 
algorithm, both doing so through processing enormous amounts of data. The 
Edmond Belamy painting later made history as ‘the first portrait generated by 
an algorithm to come up for auction, and eventually sold by Christie’s for a not 
insubstantial sum of US$432,500 (Cohn 2018). The key issue in these cases is 
that the algorithms have not been specifically programmed to place notes and 
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paint; instead, they ‘learnt’ to do so through processing enormous amounts of 
data and being given signals on what placements were ‘correct’ or ‘wrong. On 
that feedback, they then generated their respective outputs. 

While not quite the spectre of a Terminator machine out to annihilate the 
human race, creative AI on these terms is disturbing in how our lack of under- 
standing of its creativity and creative process reinterrogates our notions of 
humanness, where creativity has always been its indisputable hallmark (Zausner 
2007). It is ‘part of what makes us human’ (Sawyer 2006, 3), and affirms our 
humanity (Csikszentmihalyi 1990); it colours the domains in which humans 
work, think, play, produce and perform (Kaufman and Baer 2005). Per the 
opening quotation of the Introduction, the clones in the speculative society of 
Kazuo Ishiguro’s (2005) acclaimed novel, Never Let Me Go, made art as a con- 
certed attempt to evidence their humanity. As their teacher explained to them: 
‘we thought it would reveal your souls. Or to put it more finely ... to prove you 
had souls at all’ (Ishiguro 2005, 238). The clones’ creative work were sought to 
demonstrate humanity, for ‘the creativity code, as Marcus du Sautoy (2019) 
puts it, ‘is a code that we believe depends on being humar (2-3). How, then, 
may we understand so critical a touchstone of humanness in AI when creativity 
is seemingly manifest on such opaque and unintelligible terms? 

This chapter thus proposes a framework of de-familiarisation for the paradox- 
ical task of rendering the computer's creativity, seemingly so entrenched on its 
own terms of computational data and processing, intelligible in human terms. 
Its aim is to propose an approach with which to rationalise the processes of the 
computational algorithm, if anything to render the clarity of the imbrications 
between the human and the computational that colour so much of our algorith- 
mic world today. First, as a brief literature review, I present a few salient tenets 
of existing rationalisations of AI. In particular, I critique how their approaches, 
by and large, extract comparative analyses between human functions and com- 
putational processes. To formulate an alternative approach, I then draw from 
rationalisations of media out of media theory, specifically theorisations of the 
marionette by Heinrich von Kleist (1810) and of the camera via Russian film- 
maker Dziga Vertov’s (1923) writings on cinema, to present a methodology of 
de-familiarisation as an approach to rationalising technology on its own terms. 
In the third section, I apply that perspective to re-think creative AI via the case 
study of AlphaGo, an algorithm programmed by Google DeepMind to play the 
game of Go and which made AI history in 2015 by becoming the first com- 
puter programme ever to beat a human professional player at the game. While 
AlphaGo does not produce artistic work per se, it serves as an apt case study 
as its moves were deemed to be of exceptional novelty - indeed, described as 
‘creative genius’ (Sautoy 2019, 34) - and in various ways considered to have 
re-defined the frontier of AI. The last section will conclude. The uniqueness 
of this argument thus lies in how it aims to shift the conversation from an us- 
and-them framework, where computing is often conceived on the singular 
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oppositional dimension of humans versus machines (such as comparing 
computers directly against humans). This alternative approach to understand- 
ing algorithms thus suggests a different dimension to that understanding - not 
one made on human terms, but as a paradoxically impossible approach of the 
algorithmic being humanly intelligible on its own terms. 


Current Rationalisations of Creative AI 


Current consideration of creative AI lies in extensive scholarship, not least 
because much of it sits within a vastly wider enquiry: can computers be human? 
In the face of this question, current discourse inevitably turns to a comparative 
methodology, whereby the computer’s processes are compared against multi- 
ple manifestations of human cognitive function, including creativity (Dreyfus 
1972; Dreyfus 1979; Bailey 1996; Moravec 1998; Boden 2004). Various conclu- 
sions are then reached by matching one against the other, and working out how 
each measures up. 

In other words, the rationalisation of AI is laid out in comparative terms, so 
that AI becomes intelligible only as against human capacities, or against what 
AI can or cannot do as compared to humans. The multiplicities which reflect 
this rationalisation across philosophy, computer science, cognitive psychol- 
ogy, cybernetics, neuroscience and myriad other disciplines are myriad and 
intricate, and far beyond the scope of this chapter to cover comprehensively. A 
few highlights will hopefully suffice to demonstrate its contours. We might, for 
instance, think about Vannevar Bush's famous imagining of a memory machine 
he named the ‘memex’ (Bush 1945), influential to the present day as a basis 
for the World Wide Web (Davies 2011). Notably, Bush presented the memex 
as a technology directly against human memory, specifically referencing the 
former’s mechanised processes of speedy and flexible consultation and stor- 
age against the latter's corresponding weaknesses, leading to impermanence 
(forgetting) and lack of clarity (confusion). Conversely, Bush also noted the 
strength and speed of association of human memory, concluding that ‘man 
cannot hope fully to duplicate this mental process [of association] artificially, 
but he certainly ought to be able to learn from it’ (Bush 1945, n.p.). Both points 
illustrate Bush’s rationalisation of technology as a counterbalance to human 
capability, whereby one variously contrasts against, supplements, and demon- 
strates differences against the other. The technology is thus made intelligible 
as against the human, specifically in terms of what it can augment and surpass, 
and what it cannot. 

As AI - itself the field of computers which simulate human cognitive 
capacities - increases in operative sophistication to resemble human intelli- 
gence, this contrast becomes ever more explicit. The Turing test (Turing 1950), 
even more famous than Bush’s memex machine, reconciled computer cognition 
in terms of whether it was distinguishable - or not - from human behaviour. 
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Indeed, John Searle, among many other computer scientists, distinguished 
between ‘strong AT and ‘weak AT in his now classic 1980 paper, ‘Minds, Brains, 
and Programs’ (Searle 1980), later developed into his book, Minds, Brains and 
Science (Searle 1984), precisely on such rationalisations of the computer against 
characteristics of human cogitation. In his paper, Searle argued that AI, in its 
state of development then, could only be ‘weak, whereby ‘the principal value of 
the computer in the study of the mind is that it gives us a very powerful tool. 
Conversely, it was not ‘strong, whereby ‘the appropriately programmed com- 
puter really is a mind, in the sense that computers given the right programs can 
be literally said to understand and have other cognitive states ... the programs 
are themselves the explanations’ (417; emphasis in original). Searle clinched his 
argument against ‘strong AT by arguing its lack of free will and other mental 
states which, according to Searle, characterise human cognition, thus demon- 
strating the limitations to ‘computer simulations of human cognitive capacities’ 
(417). Again, these arguments rationalise AI against the human in a compara- 
tive mode. They render Al intelligible by referencing its technological Other- 
ness against constructed definitions of natural human responses. 

As a counter-stroke, we might also think about the extensive work in com- 
puter and cognitive sciences which rationalise the human being in computa- 
tional terms. However, this shifted understanding of the human as a computer 
only expands the commensurability between AI and humans, this time not by 
difference (humans against computers), but by equivalence - humans are com- 
puters. In turn, this intelligibility of AI via counterpointing the human - in 
terms of underscoring ATs logical and mechanised processes as against the bio- 
logical and the organic - expands to not only the rationalisation of the human, 
but the world itself. This, then, is the core of computationalism, defined by 
Golumbia (2009) in its ‘received’, or ‘classical’ form, as 


... the view that not just human minds are computers but that mind itself 
must be a computer - that our notion of intellect is, at bottom, identi- 
cal with abstract computation, and that in discovering the principles of 
algorithmic computation via the Turing Machine human beings have, in 
fact, discovered the essence not just of human thought in practice but all 
thought in principle (emphasis added). (7) 


The idea has flexed and flagged in multiple ways and forms, but its central con- 
cept remains the conceptualisation, understanding and identification of human 
cognition and mind in computational terms. In this respect, we can also think 
about, for instance, Allen Newell and Herbert Simon’s work across the decades 
from the 1950s which specifically argued for the model of all human reason- 
ing to be representable as symbolic ‘information processing systems’ (Newell 
and Simon 1972). Giants in their respective fields, both awardees of the Turing 
Award and Simon as well a Nobel laureate, their thinking converged with others 
at the Dartmouth summer conference of 1956, whose specific mission, notably, 
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is ‘to proceed on the basis of the conjecture that every aspect of learning 
or any other feature of intelligence can in principle be so precisely described 
that a machine can be made to simulate iť (McCarthy et al. 1955, n.p.). 
As the field (and the term) of ‘Artificial Intelligence’ emerged out of the Dart- 
mouth conference, its ideas folded into another new field developed in the 
1970s - namely, cognitive science, which studies human thinking, learning and 
perception as coloured by cybernetics, neuroscience, linguistics and psychol- 
ogy, but also dominated by AI, mathematics and computation (Gardner 1985). 
The field of cybernetics, first emerging out of the Macy Conferences on Cyber- 
netics from 1943 to 1954, also forged new paradigms out of information theory, 
neural functioning and computer processing, among others, to become ‘a new 
way of looking at human beings. Henceforth, humans were to be seen primar- 
ily as information-processing entities who are essentially similar to intelligent 
machines’ (emphasis in original; Hayles 1999, 7). 

This approach of rationalising the human in computational terms infuses 
much of current thinking about creative AI (Boden 1996; Boden 2004; Miller 
2019; Sautoy 2019; Kaufman and Baer 2005). The definitional knottiness of 
the term ‘creative’ aside - over 60 definitions of ‘creativity’ appear in psycho- 
logical literature alone (Boden 1996, 268) - the broad rationalisation of crea- 
tive AI continues along comparisons against human creativity as couched in 
computational terms. Hence, for instance, Douglas Hofstadter suggests the 
‘mechanisation of creativity’: while ‘creativity is the essence of that which is not 
mechanical, ‘[y]et every creative act is mechanical - it has its explanation no 
less than a case of the hiccups does’ (emphasis added; Hofstadter 2000, 669). 
Similarly, Herbert Simon, as already seen, justifies human cognition as infor- 
mational processing systems, and thus posits that ‘creativity involves nothing 
more than normal problem-solving processes’ (as quoted in Csikszentmihalyi 
1988, 19). More recently, Miller (1992; 2000; 2019) rationalises human creativ- 
ity as ‘a model for network thinking’ in terms of ‘many lines of thought taking 
place at once in parallel, coming together from time to time to enrich each 
other’ (Miller 2019, 36). The model thus demonstrates how ‘new ideas do not 
just pop up out of nowhere, even though they may seem to, thereby visualising 
human creativity as a mappable process that is also reproducible on a com- 
puter (Miller 2019, 29). Having said all that, the idea of mechanised creativ- 
ity stretches back to the ancient Greeks, where Burkholder, Grout and Palisca 
(2014), for instance, argues that Pythagoras and his followers held that ‘num- 
bers were the key to the universe; and thus thought music as ‘inseparable from 
numbers’ (13). Creating music, then, was really the theoretical application of 
numbers and various mathematical properties in logical and calculable steps, 
not unlike in algorithmic fashion. 

Across these formidable rivers of thought and expansive arguments, if 
sketched in generous outlines and overlooking many more, we may thus iden- 
tify how rationalisation of human creativity along mechanical computational 
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processes dovetails into the persistent thinking of AI as made intelligible 
against the human. As such, the intelligibility of AI continues to be tracked 
against shifting interpretations of human capability and, in that respect, facile 
notions of what constitutes humanness. The arguments which directly oppose 
creativity as defined in terms of logic and mechanisation just as easily deploy 
a notion of creativity that appeals to other touchstones of humanness deemed 
still unachievable by the computer, such as consciousness, self-understanding 
and awareness: a truly creative computer, after all, cannot be a dumb savant 
that naively flings outputs at an audience’ (Veale and Cardoso, 2019, 4). Or that 
creativity entails unique human experiences, such as ‘the need for experience 
and suffering’ (Miller 2019, 16) or self-actualisation, where creativity is ‘about 
humans asserting that they are not machines, and ‘to expose what it means to 
be a conscious, emotional human being’ (Sautoy 2019, 283). Hofstadter (2000) 
refers to ‘the depth of the human spirit’ for a meaningful notion of creativity: 


A ‘program’ which could produce music as they [Chopin or Bach] did 
would have to wander around the world on its own, fighting its way 
through the maze of life and feeling every moment of it. It would have 
to understand the joy and loneliness of a chilly night wind, the longing 
for a cherished hand, the inaccessibility of a distant town, the heartbreak 
and regeneration after a human death. It would have to have known 
resignation and worldweariness, grief and despair, determination and 
victory, piety and awe. (672-673) 


The point here is not to argue for any particular definition of or position about 
the computer's creativity. Rather, it is to underscore the various lenses deployed 
through which AI creativity is rendered intelligible, namely, in relation to the 
human in terms of comparison, contrast and analogy. In some ways, this is 
wholly intuitive - as mentioned, technology forms a counter-distinction to 
humanness; it mirrors the age-old binary of artifice against nature. It thus 
makes sense to employ humans as the referential framework in understanding 
the technological Other. Yet, the approach is also flawed. Understanding AI 
on these terms becomes subject to changing constructions and perspectives of 
humanness, so that it relies on a precarious balance of what AI is against what 
it is not. Conditional on being defined in relational terms, it fails to have its own 
definitional footing. An understanding of AI on these foundations cannot be a 
thorough one. 

Moreover, this approach of comparison confines our understanding of AI to 
being within the intelligibility of human terms, rather than made intelligible on 
its own terms as a logical, mechanical and computational entity. This is impor- 
tant because at the heart of an intelligibility on human terms is an incommen- 
surability that is never truly addressed: computers are simply not humans. A 
comparison that renders one intelligible on the terms of another will always 


56 Al for Everyone? 


lose something in the translation. The next section, drawing on alternative per- 
spectives from theorisations of media, will suggest a different approach. 


De-familiarising Creativity 


As a starting point, much of media technology is similarly rationalised 
as comparisons against human capabilities, as this broad scattering of examples 
will hopefully suffice: Nicholas Carr (2010), for instance, in resonant echoes 
of Marshall McLuhan ([1964]; 2013) rationalises technology as an expansion of 
‘our power and control over our circumstances (44), such as the map and the 
clock which ‘extend and support’ the ‘mental powers’ of humans in formulat- 
ing, producing and sharing knowledge. Jonathan Safran Foer (2016) writes of 
communication technologies as ‘substitutes’ for real-time face-to-face human 
interaction: “We couldn't always see one another face to face, so the telephone 
made it possible to keep in touch at a distance. One is not always home, so the 
answering machine made a message possible without the person being near 
their phone? (n.p.). Edward Branigan (2006) suggests anthropomorphism as 
an ‘analytic category’ to measure ‘the degree to which a camera is being used to 
simulate some feature of human embodiment; whose analysis then relates the 
qualities of the camera to ‘a typical way of human viewing, or moving (or 
thinking and feeling), and to what degree’ (37). William Brown (2009) argues 
the pole converse in his thinking about ‘posthumanist cinema, demonstrating 
how the digital ‘posthuman camera omits human embodiment entirely in its 
humanly impossible shots. 

This comparative approach is also an old rationalisation. In his 1880 essay, 
Jean-Marie Guyau analogizes the human brain to the phonograph, drawing 
connections between recorded sound as grooves of vibrations engraved onto 
the phonograph’s metal plate and the ‘invisible lines [that] are incessantly 
carved into the brain cells, which provide a channel for nerve streams’ (31). 
Guyau further analogizes the speed and strength of vibrations of our brain cells 
in terms of images conjured by our minds to the speed of the phonograph’s 
vibrations and the tones of its sounds. In 1950, George Wald, a professor of 
biology at Harvard, noted resemblances between the camera and the human 
eye - ‘of all the instruments made by man, none resembles a part of his body 
more than a camera does the eye’ (32). Wald (1950) further detailed similari- 
ties between human vision and photography: ‘the more we have come to know 
about the mechanism of vision, the more pointed and fruitful has become its 
comparison with photography’ (32). He described how the chemical changes of 
exposed photographic film, particularly ‘dark reaction of the ‘latent image’ in 
the darkroom, mirrors processes of vision in the eyes exposure of rhodopsin’ 
to light (40). Along these broad contours, the rationalisation of media technol- 
ogy thus echoes that of AI - as matched against human capabilities, reflecting 
similarities and differences; as situated in human terms. 
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However, large swathes of critical theory have grown to critique the human as 
a referential framework. Specifically, this work decentres the human by shifting 
the critical lens to those that are not human, such as ‘understood variously in 
terms of animals, affectivity, bodies, organic and geophysical systems, material- 
ity, or technologies’ (Grusin 2015, vii). Understanding the Other and accom- 
modating on their terms their complex involvement in the consideration of our 
world thus stands as a long-established enquiry through the humanities. This 
work spans across multiple areas, such as the social imaginaries of inanimate 
objects (Appadurai 2014); the posthuman (Braidotti and Hlavajova 2018); the 
nonhuman (Grusin 2015); post-anthropocentrism (Parikka 2015); the intelli- 
gibility of cinema as both subject and object of vision (Sobchack 1992) - to cite 
again just a few sprinklings as illustration. There are many others. 

There is, of course, an inherent contradiction to this approach, which is of 
intelligibility having to be made intelligible in alien terms, or in terms of an 
Other-ness that, by definition, we do not and are unable to possess. How might 
we render something intelligible in its own terms if it is, by definition, outside 
the intelligibility of our own terms? How do we accommodate our understand- 
ing around something that we are not, let alone fathom its terms? It is certainly 
a valid conceptual difficulty. The key is to understand the enquiry not as a literal 
one which seeks literal answers. Rather, it is one which involves speculation and 
imagination in envisioning the perspective of the Other as part of its methodol- 
ogy. It requires the acceptance of the philosophy of things being in themselves, 
beyond and independent of our experience (Moffat 2019). It entails being open 
to indeterminacy and contingency, and of acknowledging the nature of things 
as based on, while not pure fantasy, nonetheless an inexact science. 

The approach proposed here, then, for shifting one’s critical perspec- 
tive in relation to an intelligibility of the technological Other is to draw on 
media theory’s alternative rationalisation of technology, namely, through 
de-familiarisation - to disturb or disorder the terms in which we think of an 
entity so as to re-learn it on different terms, specifically those of its own. I 
underscore two examples, each of a different media through different ration- 
alisations, to more fully illustrate this approach. The first is Kleists 1810 essay 
on marionettes, in which he recounts a conversation with his friend, ‘Herr C} 
who expressed admiration for the gracefulness of the puppets. This position is 
counterintuitive: puppets, controlled by their puppet-master, are mechanical and 
lifeless; it is senseless to consider puppets as graceful as, if not more graceful 
than, human dancers. The key in Herr C’s’ reasoning lies in how he re-reads the 
puppet’s mechanical movements not as cold actions with no consciousness or 
with a surrendered volition, but as precisely the nonconscious and mechanical 
movements that only puppets are capable of, through which beauty and grace 
re-emerges. The puppet’s artificial properties are thus read on their own terms - 
not against the human dancer’s consciousness of its movements which ren- 
ders their artistry and beauty. Rather, “Herr C? de-familiarises what and how 
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we think about grace, and relearns it in the non- or unconsciousness of the 
puppet’s mechanical operations. We thus come to a different understanding of 
the puppet by emerging on the other side of its paradox (i.e., of grace from the 
controlled and the automatic) to arrive at an alternative intelligibility of it and 
its movements. We understand the puppet on its own terms. 

The second example is drawn from Russian filmmaker Dziga Vertov’s theori- 
sation of cinema. Articulated in the 1920s through various pamphlets, articles, 
manifestos and public addresses, and fresh from radical societal change in the 
wake of the Russian Revolution, Vertov sought from cinema and its camera 
the newness of humanity and society. He read the camera through a new intel- 
ligibility - not against the human or in comparison to the human eye, but on its 
own terms as what he calls a ‘kino-eye’ in how it sees a different world: I am a 
mechanical eye. I, a machine, show you the world as only I can see it’ (emphasis 
added; Vertov 1923, 17). Of course, the human hand and eye still control the 
film camera; it has not literally come alive. But the argument here is not a literal 
one. Rather, it is a theoretical shift involving imagination and inventiveness to 
acknowledge the new-ness of the camera’s vision and its alien visuality. Like 
Kleist with the puppet, Vertov came to understand the camera on its own terms 
as he sought a visualisation of the new society birthed from revolution out of 
its camera eye: its alien-ness as an un-human consciousness is precisely why 
the kinoeye is capable of ideology to present the real in a way the human eye 
cannot. He could thus acknowledge the camera’s different-ness - as he writes, 
“it is the realization by kinochestvo* of that which cannot be realized in life’ 
(Vertov 1922, 9). Hence, through re-reading the camera on its own terms, Vertov 
de-familiarised the world around us, presenting it anew in a radical language 
borne out of the camera’s foreign intelligibility, and shifting images out of the 
referential framework of human seeing. 

The point here is not to agree or disagree with Vertov or Kleist in their respec- 
tive readings of media. It is to illustrate how a frame of reference in under- 
standing can shift with a different rationalisation, and in so doing recognise a 
different intelligibility. The task, then, is to apply this approach to understand- 
ing creative AI on their own terms, to which the next section will turn via the 
case study of AlphaGo. 


Rationalising the Creativity of AlphaGo 


AlphaGo, an algorithm programmed to play the game of Go, achieved global 
fame in March 2016 by defeating Lee Sedol, a highly ranked South Korean pro- 
fessional Go player and 18-time world champion, 4 games to 1 in a 5-game 
tournament. Its victory sent ripples through the AI community and the wider 
public because ‘teaching computers to master Go has long been considered a 
holy grail for artificial intelligence scientists’ (Yan 2017, n.p.). The difficulty of 
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this ‘holy grail’ lies in the game’s high level of abstraction. Played by two players 
each placing, in turns, stones of their respective representative colour (black or 
white) on intersection points between horizontal and vertical lines marked on 
a board, there are essentially just two rules of play: one on how to ‘capture’ an 
intersection; the other on how that intersection is considered ‘occupied. The 
goal is simple: to have, at the end of the game when all intersections have been 
‘occupied, stones of your colour ‘occupy’ more intersections (or territory) than 
your opponent’. 

Like all good Zen koans, its minimalism is also its complexity. Compared to 
Go, chess, as a fellow strategy game, is clearer in various ways: fewer moves can 
be made to start a chess game, and thus relatively fewer possibilities branch out 
from each opening move. Pieces also have set values (the pawn, for instance, 
has the lowest; the queen the highest) which makes an unfinished chess posi- 
tion relatively easy to calculate and analyse as to which player is winning based 
on how many and which pieces are left, plus any positional advantages. In com- 
parison, because all there is to Go are stones on line intersections, the result 
is many more possible board configurations, each one lending themselves to 
even more possible positions if calculated further down the line. As a result, 
there is quantitatively and qualitatively more ambiguity in Go, with ensuing 
greater difficulty in analysing who is winning from an unfinished game posi- 
tion. Hence the significance of AlphaGos victory: due to its multiple positional 
possibilities - as has been oft-quoted, there are ‘more possible configurations of 
the board than there are atoms in the universe’ (Yan 2017, n.p.) - until Alpha- 
Gos triumph, the game was considered unconquerable by computers over 
human players simply because its level of complexity needed it to be played 
with human abilities of intuition and grasping of visual structure, rather than 
the computer’s powers of searching and calculation of variations. 

Pertinently for our purposes here, AlphaGo was vaunted for not only its 
tournament victory, but also the creativity of its moves. One move - Move 37 of 
Game 2 - in particular was so wholly unexpected that commentators described 
it, if a tad gushingly, as ‘a truly creative act’ (Sautoy 2019, 37); or one of the 
most creative [moves] in Go history’ (Tegmark 2017, 89). AlphaGo’s Move 37 
was to place a stone on an intersection on the board’s fifth line, a move very 
seldom played at that stage of the game because it was considered too ‘high 
on the board, giving the opponent room to play on the fourth line down and 
thereby gain too much solid territory. The media quickly attributed the move 
to the algorithm's own creativity, lauding it, with embarrassing hyperbole, as 
‘the move no human could understand’ (Metz 2016, n.p.). Or, as widely quoted 
from Fan Hui, the European Go champion who was the first professional Go 
player to play and lose against AlphaGo: ‘it’s not a human move. I’ve never seen 
a human play this move! (as quoted in Metz 2016, n.p.).” What validated the 
unusualness of the move - and thus rendering it ‘creative’ rather than ‘insane’ 
or ‘nonsensical’ - was that, some fifty moves later, that fifth-line stone became 
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an unexpected linchpin to a battle for territory which started in a different part 
ofthe board. In due course, the battle joined up with the Move 37 stone, giving 
AlphaGo the advantage and eventually the win. 

As expected, the rationalisation of AlphaGos Move 37 lay in the conven- 
tional framework of human terms via comparison and contrast, this time by 
placing the algorithm in its own intelligibility as one outside human sense. Yet 
that does not achieve much for understanding AI in its conceptual sense - 
it merely blankets the algorithm with mystique of the technological and a 
cryptic referencing of its Other-ness on the basis of some kind of mysteri- 
ous agency. For instance, much was made of AlphaGo’s independent learning 
from its neural network to generate its moves. Like the painting of Edmond 
Belamy whose generator network ‘learnt’ paint placement from the discrimi- 
nator network, AlphaGo, as the generator network, ‘learnt’ the best moves in 
Go by playing multiple games against another neural network. While the dis- 
criminator network would have been ‘trained’ to play Go by being fed (by 
human computer scientists) millions of games (played by human players) as 
downloaded from the internet, it is the processing of the millions upon mil- 
lions of games between AlphaGo and its discriminator network that makes up 
AlphaGos main ‘training, namely, the calibration of the values and weight- 
age for its nodes across its various networks which ultimately generates 
AlphaGoss moves. 

The implication, then, is that the algorithm's creativity in coming up with 
unusual moves is its own, generated on its own steam and out of its own 
learning, an idea its Google DeepMind creators were keen to perpetuate. For 
instance, in a video interview with CNBC, Demis Hassabis, CEO of Google 
DeepMind, implies the same generative creativity, explaining how algorithms 
such as AlphaGo ‘learn from scratch, learn from their own mistakes ... they 
learn from themselves, directly from data or from experience, rather than being 
told what to do by human programmers (emphasis added; as linked in Yan 
2017, n.p.). DeepMind have since developed AlphaGo’s successive algorithms, 
AlphaGo Master and AlphaGo Zero, along similar lines, namely, to ‘learn’ Go 
rules without any human guidance, but simply through processing millions and 
millions of games against another neural network, whose ‘reward’ and ‘punish- 
ment’ outcomes would thus train the algorithm on the rules and optimisation 
of gameplay (Silver et al. 2018). 

The case here, then, is to re-think AlphaGos creativity on its own terms, as 
with Vertov and Kleist’s up-ending of grace and visuality in relation to mari- 
onettes and cinema. Here, we re-orientate the thinking of AlphaGo’ creativity 
from its comparisons against moves by human players to de-familiarise its crea- 
tivity so as to stand on its own terms. Move 37 was not generated on non- or 
unhuman terms as an alien stroke of creativity; it was calculated out of multitu- 
dinous values and possibilities arising from that particular position, and then 
chosen as the one which gave it the highest chance of ending up with more 
territory and thus a win. 
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But ‘creativity’ here, as framed on the algorithm’s own terms, is not only its 
multi-layered® levels of calculation of the multitudes of moves from the multi- 
tudes of board positions to the multitudes of possible future board positions. 
Such level and extent of calculation resonate with the earlier discussion 
on creativity as an account of mechanised thought and logic applicable to 
human cognitive systems. There is a further nuance here: the algorithm's sense 
of creativity does not just lie in this manifold expansion of logical thinking 
(which might indeed be traced back to human interventions in training the 
discriminator network); it is also about the speed of its calculation through 
the multitudes of board position data. Speed, then, is really about space, or the 
demolition thereof, a la Paul Virilio (1991) who calls speed ‘a primal dimension 
that defies all temporal and physical measurements’ (18), and which directly 
results in ‘the crisis of the whole, whereby the substantive, homogeneous and 
continuous gives way to the fractional, heterogeneous and discontinuous. 
Without veering too much into Virilios ideas on speed which include the city, 
urban architecture and media, we can draw this line of thought on speed and 
space back to the de-familiarisation of creativity, whereby the terms of the algo- 
rithm are thus about neither its anthropomorphised independent learning nor 
even its mechanisation of logic and thought. Rather, they are about the algo- 
rithm’s speed as the fracturing of the space-time of thought and as mirrored by 
the multiple splitting of its tree searches that is key to its algorithmic operation, 
and further constantly mapped with the algorithm’s training from its datasets. 
Thus de-familiarised, we can shift our conceptualisation of creativity from cog- 
nitive processes in human terms to a different framework of space and spatial 
dimension across which the algorithm traverses with speed. The more tightly 
controlled the space is with unambiguous rules and outcomes, the more it is 
suited for discontinuous and fractional spaces, and the more the algorithm will 
thrive. The rationalisation of its ‘creativity’ in generating brilliant moves with 
large probabilities of achieving game-winning positions is thus not based on 
the depth of its logical thinking and learning as per the terms of rationalised 
human creativity. Instead, based on its computational calculatory processes, we 
can read it as a more de-familiarised conceptualisation of space, and begin the 
course of understanding it on its own terms. 


Conclusion 


The man-machine assemblage varies from case to case, but always with 
the intention of posing the question of the future. 
- Gilles Deleuze (1985, 263) 


Intelligent AI - more specifically, runaway intelligent AI which not only sur- 
passes humans in general intelligence but whose capabilities are no longer 
under human control - has been identified as an existential risk, capable of 
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wiping out the human species (The Economist 2020b; Bostrom 2014). Stephen 
Hawking has pronounced to the BBC that ‘the development of full artificial 
intelligence could spell the end of the human race’ (Cellan-Jones 2014, n.p.). 
AI represents profound fears - culminating in our extinction - but also 
profound hopes in bettering life for humanity and life on Earth. 

For these reasons - to ward off our fears and harness AI for betterment - 
the need to continually push for deeper understanding of AI is also corre- 
spondingly clear. The current rationalisation of AI persists in human terms, 
as is evident from even the most recent musings on the limitations of AI (The 
Economist 2020a): comparisons are consistently made with human learning 
and cognition, such as ‘embodied cognition, or references to the ‘irredeemably 
complex’ nature of human minds (n.p.). In filling the gap of understanding why 
AL is still rubbish at doing elementary tasks that humans accomplish without 
much thinking, such as recognising a stop sign, the current approach appears 
to be to improve machine learning by developing it to resemble human learn- 
ing; to write an algorithm that edges ever closer to human cognition, namely, 
achieve the dream of ‘strong’ Al. 

But perhaps that is neither the question to ask, nor the appropriate task at 
hand. What is ultimately still not completely explainable is how algorithms 
think. And while this appears to be a technical question - to open the ‘black 
box’ - there are also other ways of arriving at an understanding to answer that 
question. John Seely Brown’s (2017) words come to mind: “We must also be will- 
ing to constantly reframe our understanding of the world. We must regrind our 
conceptual lenses, and regrind them often. (n.p.). One of these ‘re-ground’ con- 
ceptual lenses, as this chapter has argued, is the issue of intelligibility, insofar as 
the task of intelligibility is to make the unknown known. In this chapter, I have 
argued for an approach to an alternative intelligibility via a conceptual approach 
of de-familiarisation, one that shifts the terms of understanding away from the 
human to those of an Other. In making this argument, I am aware I have scythed 
through whole swathes of literature, if only hoping to at least demonstrate the 
broad contours of the argument. It is also clear that much more work needs to be 
done to hone this approach into a systematic methodology of a robust concep- 
tual framework of intelligibility applicable to algorithmic systems. But the first 
step, at least, is taken in attempting to frame an alternative question. For, per this 
sections opening quotation by Deleuze, the question of our intelligent machines 
in human society is not about the technology, but always of our future. 


Notes 


! Randomness has also been long associated with creative work by humans, 
such as aleatory poetry or music by the Surrealists. 

> More fully known as the Dartmouth Summer Research Project on Artificial 
Intelligence. 


An Alternative Rationalisation of Creative Al by De-Familiarising Creativity 63 


> A pigment containing sensory protein which, for many seeing animals, 
including humans, is located in the retina of the eye that converts light into 
an electrical signal. 

This is a neologism coined by Vertov, referring approximately to ‘the 
quality of the cinema-eye, as noted by the editor and translator of Kino-eye 
(Vertov 1984). 

It should be clarified that, contrary to the medias hyperbole, human play- 
ers have indeed played a fifth line move before and to productive results, 
as part of the move's strategy would be to emphasise influence and speed 
to battle for more centre territory in return for giving up peripheral terri- 
tory, akin to a chess gambit of giving up a pawn piece for centre control. 
The more precise reading here might be that the context for that strategy 
(for centre territory) was not present in this game, which was what made 
AlphaGo’s move so strange and thus creative, rather than to claim that it 
was an ‘inhuman move. 

There are three networks to the algorithm: the policy network, which ‘come 
up with what would be the interesting spots to play’ to build up ‘a tree of 
variations the value network, which ‘tells how promising is the outcome 
of [each] particular variation, and finally the tree search, which would look 
through different variations and ‘try to figure out what will happen in the 
future. AlphaGo - The Movie, 47:15-47:50. 
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CHAPTER 5 


Post-Humanism, Mutual Aid 


Dan McQuillan 


Introduction 


There is a growing awareness of the pitfalls of applying AI and algorithms to 
important social problems. Machine learning can only learn from past data and 
it’s pretty clear that means a perpetuation of existing biases. This collision of AI 
with civil rights has led to corrective efforts at both technical and ethical lev- 
els (Feldman et al. 2015), (High-Level Expert Group on AI 2019). Meanwhile, 
other observers have pointed out the ways that AI adds its own asymmetries 
to an already skewed social landscape (Eubanks 2018). Theres more data about 
the poor and marginalised because they are already most surveilled, and they 
are most surveilled because our social systems already categorise them as trou- 
blesome. As a result, any unfairness that algorithms add to the mix will fall 
more heavily on those who are already struggling the most. However, it’s not 
only or even mainly data that shapes the politics of AI. 

Langdon Winner wrote about the way particular technologies appear to 
have an inherent compatibility with particular socio-political systems (Winner 
2020), so it’s fair to ask what feedback loops connect AI and the societies into 
which it has emerged. This attentiveness may help to bring neglected features 
to the fore, to remind us of framings that are so pervasive they are usually 
ignored or to highlight new dynamics that are going to change more than just 
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our means of technical ordering. For the purposes of this chapter it is impor- 
tant to ask these questions not only to provide a bigger picture of the problems 
of AI, but to predict the problematic nature of the likely reaction to it. The con- 
tention is that a reactive if understandable response to the harms caused by AI 
will itself risk feeding into wider crises instead of escaping them. The first step 
in unpacking this is to be more concrete about AI and about the resonances 
that are set up between its mathematical logic and its action in the world. 


The Logic of AI 


Actual Al is a form of machine learning; that is, an approach to computational 
problem-solving that iterates to a solution using data. It’s different to more 
traditional forms of computational modelling: instead of trying to simulate the 
inner workings of a system, it’s a transferable method of number crunching that 
simply requires sufficiently large amounts of training data. Its also different 
to traditional statistics, although it branches off from that family tree - where 
statistics tries to assess very precisely the relationships between variables and 
the robustness and possible error in the parameters, machine learning really 
doesn't care - its only goal is to make repeatable predictions. Whereas sta- 
tistics is realist (in trying to model an underlying truth), machine learning 
is instrumentalist. 

These may seem like nerdy distinctions but they have major consequences 
when it comes to social impacts, not least because of the inherited aura of infal- 
libility that machine learning inherits from its associations with science and 
statistics. Like them, it’s an approach that elevates quantitative analysis over 
any other form of insight. But machine learning is all about prediction and not 
about explanation. For machine learning, all that matters is generalisability; 
does the pattern learned from training data perform well on test data, in which 
case it can be let loose on the world. 

When people talk about practical AI they mean machine learning as number 
crunching, and not any of the symbolic attempts to seriously emulate human 
reasoning that used to be called ‘strong AT. Even the term ‘learning’ has, at dif- 
ferent times, meant a more profound attempt to understand the way we learn 
as embodied beings with life experience (Marcus 2018). But these approaches 
struggled to produce practical results, whereas the form of machine learning 
that simply means improving with ‘experience’ (i.e., with data) has succeeded 
spectacularly at previously impossible tasks. If current machine learning has a 
psychological analogue it is Skinner’s behaviourism, where observable behav- 
iours supersede introspection or any understanding of motivation in terms 
of meanings. 

The form of machine learning which most accelerated the current ‘AI revolu- 
tion is the artificial neural network, which symbolises all these important ten- 
dencies more vividly than any other. To begin with, a neural network sounds 
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like something to do with the brain, and while it’s true that biological neurons 
were the original inspiration for the computational neurons that are densely 
interconnected in the layers of so-called deep learning, they are nowadays 
understood as arrangements constructed purely for predictive efficacy and 
not because of any residual correspondence to organic brains. Each neuron 
sums the weighted inputs from those in the previous layer that are connected to 
it, then applies an ‘activation function’ to the signal it passes to neurons in the 
subsequent layer (Nielsen 2015). Deep learning depends on orders of magnitude 
more training data than other methods of machine learning and its number 
crunching is on a previously inconceivable scale. The weights at each neuron 
are varied by the optimisation algorithm, and the optimal set of weights are 
become the ‘model’ that has been learned. The inner operations of a neural 
network are more opaque than other machine learning methods, making it 
extremely difficult to unravel their workings and understand in detail how they 
reached their conclusions (Feng et al. 2018). 

The mathematical logic sets out to ‘learn’ by minimising a loss function; 
loosely, the difference or distance between its current predictions and the 
training data. This requires a well-defined objective to optimise on, which is 
typically the target classification of interest. The formalism of machine learning 
expresses any worldly context in terms of a fixed set of possible outcomes which 
are functions of the input features of each instance of data. Taking the world to 
be at least functionally composed of entities and their attributes is a philosophi- 
cal commitment to an ontology of separate individuals and objects, while the 
very idea of optimisation is itself value-laden; AI promotes a market-friendly 
and mathematised utilitarianism. 

The mathematical iterations of machine learning are implacable in their 
pursuit of the assigned outcome, so harnessing them to a messy social goal 
inevitably sets the stage for shockwaves of unintended consequences. Given 
the requirement for the context of interest to be expressed as purely numerical 
(and measurable) abstractions, it is also inevitable that the outcome being opti- 
mised on is itself a distant proxy for the desired policy goal (Malik 2020). For 
machine learning, the external environment is a fixed set of givens; woe betide 
those who rely on it when the underlying distribution shifts (even though it 
is the nature of the world to constantly change). AI is haunted by the under- 
examined constructs it assumes in order to make the world amenable to its 
methods; above all, that the world is essentially a mechanism which can be 
manipulated by the adjustment of its parameters (Wu 2019). 

Al is undoubtedly successful at tackling data sets which were previously off- 
limits in terms of scale and complexity. Ignorance is strength; by bypassing 
the need to explain and moving straight to prediction, AI provides a ready- 
to-hand tool for intervention. Never mind that correlation is not causation; that 
explainable is not the same as explanatory (even if I can tell which combina- 
tion of parameters is most significant in determining the final classification, it 
doesn’t provide me with a causal understanding). The predictions of AI are a 
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dilution of science because they are not the expression of a hypothesis but sim- 
ply an extrapolation from a set of data points. And yet AI is performative; by 
intervening in the world and reorganising the phenomena it claims to describe, 
it brings our experience into closer alignment with its constructs (Mackenzie 
2008). This invalidates the methods as a form of statistical insight but makes it 
very effective as a mode of subjection. 


Automated Segregation 


As the logic of AI migrates from the abstract mathematical space of tensors 
to the space of real social tensions, it comes to bear in specific ways. First and 
foremost of these is automated segregation. 

There is nothing personal about the predictions of AI - at root, they are always 
some form of labelling in terms of ‘people/objects like you. As an offshoot of 
the statistics family tree, machine learning’s classifications are governed by the 
heavy hand of the central tendency (Malik 2020); that is, the principle that 
there exists a central or typical value for a probability distribution. Predictions 
about you are centred on some recomposition of the past behaviours of those 
with similar attributes. Such a prediction may be useful for an institution deal- 
ing with large numbers of people at a distance, but it is not about you at all. The 
subjects of AI are represented as entities with attributes inasmuch as they are 
present to the algorithm as vectors of values. 

Understanding how this plays out in terms of the distribution of benefits 
and harms means reflecting on resonances between the intrinsic logics of 
AI and our social institutions. When these algorithms execute mathematical 
operations of classification, ordering and ranking that carry over into our lived 
experience, they offer support to certain ways of doing and limit the likelihood 
of certain others. The significance of these resonances will vary with context. 
It doesn't seem problematic to classify and rank the likely failure modes of an 
engineering infrastructure but it becomes far more delicate the closer we apply 
the same approach to other people. Questions of class and classification, the 
assumption of certain orders as normative, and ideas about rank and hierarchy 
are so deeply embedded in our psyches and societies that calculative methods 
with the same logic act as an amplifying stimulus. 

When we deal with social classification we can’t escape questions of power. 
The distribution of power in society may be complex and multivalent but it is 
also highly asymmetric. This not a problem created by machine learning, of 
course, but machine learning was produced within these structures of power 
and it is acting back on them. While the mathematics of AI may be expressed 
as matrices, it is a human activity that is inescapably immersed in history and 
culture. AI acts as an activation function for specific social tendencies. As an 
idea, or ideology, AI seeks to escape association with these worldly concerns by 
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identifying with a pure abstraction and a neoplatonic purity of forms (McQuil- 
lan 2017), but this is only a plausible cover story to those who occupy an already 
privileged standpoint. The optimisation functions themselves will in general 
also be defined from these positions of social privilege. When Al talks in terms 
of ‘models’ it means the learned weights in a neural network, not the classic idea 
of a model that describes inner workings; as the goal is prediction not explana- 
tion, it is not supplying insights that could be used for causal interventions. 

For ATs impacts on the ground, the operative concerns are discrimination 
and segregation. AI is a racist technology, in the sense that AI operates so as to 
segregate, and racism itself can be understood as a technology of segregation 
(Lentin 2018). This is easy to see when it comes to facial recognition, one of the 
most egregious applications that AI has so far gifted to society. It is not just that 
facial recognition seems to perform less well on people of colour, it is that it 
carries out what Simone Browne calls ‘digital epidemermalisation: ‘the exercise 
of power cast by the disembodied gaze of certain surveillance technologies ... 
that can be employed to do the work of alienating the subject by producing a 
“truth” about the body and one’s identity (or identities) despite the subject’s 
claim’ (Browne 2015). In other words, AI’s operations of facial classification are 
actually reconstructing the category of race for subsequent intervention (Stark 
2019). Facial recognition itself forces race onto a face. Clearly, any alternative 
approach to AI must be at the very least decolonial. When applied to people, 
ATs operations with entities and attributes distil us down to innate differences. 
It excludes perspectives from critical race studies which might question the 
construction of identity gradients, nor does it acknowledge any sociological 
understanding of why people might be trapped in particular social patterns. 

On this basis, we can confidently say that the overall impact of AI in the world 
will be gendered and skewed with respect to social class, not only because of 
biased data but because engines of classification are inseparable from systems 
of power. Writers like Virginia Eubanks highlight some of the ways this comes 
to pass for social class as well as race; how it seems to always be the poorest 
and most marginalised who bear the brunt of collateral damage from algorith- 
mic systems even when the bureaucrats involved are making sincere efforts 
to be fair (which they often arent) (Eubanks 2018). The data demands of AI 
mean that the pattern of having to trade private personal information for ser- 
vices will become even more invasive. The optimisations of AI act as an inverse 
intersectionality, applying additional downward pressure on existing fissures in 
the social fabric. Like Eubanks, we should be asking what specific forms these 
fractures will take, and how to recognise them. One marker will be the emer- 
gence of machinic moralism. The more that AI is seen as a solution to austerity, 
the more its classifications and rankings will be enrolled in the rationing of 
goods and the assigning of sanctions. AI will be put in the position of decid- 
ing between the deserving and the undeserving. The most advanced forms of 
computation seem destined to re-enact a Victorian morality. 
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We can expect a lot more ‘production of truth ... despite the subject’s claims, 
as Browne puts it. Not only digital epidermalisation but epistemic injustice, a 
concept developed by philosopher Miranda Fricker through her analysis of the 
ways women have historically been degraded both in terms of the credibility 
of their testimony and in their very capacity as a knower (Fricker 2009). The 
operations of AI at scale will produce ‘continuous partial states of exception’ 
(McQuillan 2015) where the access to shared social, political and economic 
rights will be adaptively suspended for particular cohorts with the justification 
(if one is ever given) that this is required for an overall optimisation of the sys- 
tem. Al itself is indifferent to unintended consequences and collateral damage, 
as its version of statistical methods dispenses with robust estimates of error. 
The effects on the people ‘in the loop; that is, operating within these systems, 
will be the production of carelessness because they won't be in a position to 
challenge the opaque but apparently empirical judgements of the system. The 
wider output will be a scaling of callousness, as the specific effects on the most 
vulnerable and least able to speak up will be occluded by the abstractions that 
drive the algorithms. 

Clearly, both the logics and the impacts of AI are rooted in ways of approach- 
ing the world that go deeper than a recent acceleration in computational 
capacity or the availability of plentiful training data. It’s important to try to 
characterise this overall stance towards the world, not only to challenge it but 
to be wary of challenging it in ways that simply make the problems worse. 


Machine Learning Modernism 


Machine learning can’t simply be summed up as the implementation of a par- 
ticular philosophy. It is an active and performative force in the world, which 
has the potential to change the conditions of thought itself. However, it can 
be useful to point out how much machine learning inherits from modernism. 
To start with, AI is a form of computation and its algorithms are expres- 
sions of computational thinking, that is, decomposition, pattern recognition 
and abstraction (Wing 2008). While the first two are the most apparent - a 
world decomposed into data and acted on by statistical pattern finding - it’s 
abstraction that most defines the character of ATs impact. AI is above all a 
mode of abstraction that allows any issue to be treated as a mathematical opti- 
misation problem. Any aspect of the world deemed relevant must be quantified 
and normalised for inclusion in this operation; in the innermost workings of 
deep learning, all data is rendered as vectors of numbers between zero and one. 
Al is deeply reductive, asserting in effect that it can predict the unfolding of a 
system in terms of those elements which can be reduced to data, and the only 
attributes of the world that count are those that can literally be counted. Unlike 
science, which at least seeks a careful explanation of how a layer of reality arises 
from the interactions of simpler elements, AI is epistemologically careless; it’s 
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not concerned about whether its reductions represent anything more funda- 
mental as long they produces repeatable predictions. 

Applied to the social realm, AI takes on the kind of reductiveness that was 
elucidated by Heidegger; that is, a reduction of being itself. The being of entities 
subject to ATs ‘enframing’ is reduced to a calculative order which ‘drives out 
every other possibility of revealing’ (Heidegger 2013). Making the social world 
into data for the benefit of AI is to convert ourselves into a standing reserve for 
optimisation and prediction. This echoes the way that dualistic metaphysics 
combined with the capitalism system reduce the natural world into raw mate- 
rial and resource. AI models the world, but only to get something out of it. 

The discourse of AI uses terms like ‘model’ and ‘representation pretty inter- 
changeably. They are used as shorthand for the nexus of feature set and algo- 
rithmic architecture which are being applied to get a result. The layers of a deep 
learning model apply successive transformations to the input feature space that 
will eventually allow it to be distilled down to required target labels. Each layer 
contains a different representation of the original data, by way of the weights at 
each node in the layer. A prominent practitioner likens it to trying to uncrum- 
ple a paper ball; each hand movement is like the geometric transformation car- 
ried out by one layer, and ‘deep learning models are mathematical machines for 
uncrumpling complicated manifolds of high-dimensional data (Chollet 2017). 

What gets easily overlooked in the intriguing detail is the form worlding that 
is taking place; what resonances that are set up by dealing with the world this 
way. Prioritising representations over presence may be necessary for model- 
ling, but it is a move that has political as well as philosophical implications. A 
fundamental operation of social power at any level is the power to represent 
someone’ reality back to them in a way that is asserted as being more real. AI 
is at the stage of having representations that are opaque and barely understand- 
able even to those who produce them, while these representations are increas- 
ingly relied on to make robust social interventions. 

Looking at the data as it is transformed through the layers evokes another 
essential aspect of AI; the imposition of equivalence. By representing attributes 
as numbers in a vector they are made commensurable whether they happen to 
represent ‘likes’ on Facebook or the oscillations of an ICU heart-rate monitor. 
The values of each feature are traded against each other in the iterations of the 
optimising algorithm as it seeks to descend to the minima of its loss function. 
AI effects the same operation of equivalence as money; rendering objective 
diversity in terms of values that can be traded against each other. 

AI is marked by the same modes of abstraction, representation and equiv- 
alence as the rest of modernity. It applies an instrumental rationality that 
subsumes social relationality and material under a single regime of equiva- 
lence which discards the incommensurable (Horkheimer and Adorno 2002). 
The single optic of optimisation admits no outside; the methods of machine 
learning are more than generalising, they are universalising. AI carries on the 
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tradition of modern thought that Whitehead criticised as ‘explaining away’ - 
by taking its abstractions as something concrete, everything that does not fit 
into the schema is denied the status of proper existence (Whitehead 1997). AI 
operates as automated segregation, in the same key as racism, patriarchy and 
the class system, applying an inevitable hierarchy of humanness to it subjects. 
The reaction to the evils of colonialism from liberation thinkers like Fanon was 
to demand colonial subjects’ rightful membership in the category of human 
(Fanon 2005). Given the promotion of such profoundly alienating and dehu- 
manising processes it is only natural that the obvious callousness of AI will be 
opposed by calls for return to human values and to a valuing of the human. Like 
the writers and activists of postcolonialism, the unhappy subjects of algorith- 
mic governance will come to demand their full membership of the category of 
humanity. It’s in this reaction, though, that further perils lie in wait. 


Reactive Humanism 


The call for a post-algorithmic humanity that leaves no-one behind needs to 
find a way to escape the legacy of humanism. Historical definitions of human- 
ness tend to carry with them the assumption of human exceptionalism; that 
is, the uniqueness of the human in relation to the animal and material worlds. 
Whereas this originally had religious roots, the secular version born out of the 
Enlightenment centred on consciousness, morality and particular notions of 
reason. Over time, these notions have deeply shaped the psycho-political land- 
scape we still inhabit. 

There's a bifurcation in our way of understanding the world that still acts to 
separate us from the world. Whatever the success of the scientific approach, 
we still seem to distinguish ourselves as having an agency and a will that is 
different from the deterministic conception of nature that science implies. The 
foundational distinction between observer and observed remains, despite 
the challenge of quantum mechanics, and still cascades into operations as mun- 
dane as those of applied machine learning. 

The bounded individualism that comes with humanism is not merely a meta- 
physical curiosity but an active factor in our political economy. Along with the 
rational consciousness of individual actors, the very concept of the separated 
individual underpins neoliberalism and classical economics. Humanism, as the 
species-separateness of humanity, also provides the logic for treating the rest of 
nature as a resource, as an externality to be plundered at will for productivity. 
And this ‘nature’ includes, of course, those people who are in whatever way 
seen as less than fully human. 

Humanism is a vector for some of the same problems that plague a modern- 
ist AI. The historically constructed idea of the human was that it was endowed 
with the ability to make moral choices. It is exactly this aspect that led Nietzsche 
to question the idea of the T as the illusion of continuity that enables morality; 
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that is, the identity that is the cause of the actions and so is deserving of reward 
or punishment. For him, the moral concept of the T is projected onto events in 
the world (Nietzsche 1998). As we have seen, Al is already projecting algorith- 
mic forms of moral attribution into its predictions, and in line with Nietzsches 
original critique this moralism acts in the interests of some rather than all. A 
reactive humanism would only modify this mode of moralising rather than 
replacing it. 

Similarly for actions that contribute to climate change; modernist Al is part 
of a wider system where extraction follows closely on the heels of abstraction, 
where everything of the world is seen as a utilitarian resource, not a fragile 
component of a self-regulating ecosystem. AI is making its own contribution 
to global warming via the exponential increase in computing required for the 
latest deep learning models and the consequent carbon emissions (Strubell, 
Ganesh and McCallum 2019). But humanism itself, as the vision of the human 
as separate and subject to special rules, is the precursor of worldviews that have 
created the possibility of the Anthropocene. 

Perhaps the most immediately dangerous aspect of human exceptionalism 
is the one linked directly to the definition of AI; the question of consciousness 
and superior intelligence. Humanism sees the spark of rational intelligence as a 
marker of uniqueness. The field of AI meanwhile, while its current best practice 
is the steamhammer of statistical prediction, still holds on to the idea that this 
narrow form of computational ‘intelligence’ is a foothill on the way to artificial 
general intelligence; that is, machines that can think like us (Hodson 2016). 
Both humanism and AI understand intelligence as something hierarchical, that 
can be ranked from lower to higher. But ranking on the basis of intelligence is 
the backbone of race science, the pseudo-empirical justification for colonialism 
and white supremacy (Golumbia 2019). Not only that, but the implicit ranking 
of human worth by IQ has been a historic justification for eugenics through 
programmes such as forced sterilisation, and is re-emerging at the time of writ- 
ing in terms of criteria for COVID-19 triage that downgrade those seen as in 
some way disabled (NoBodyIsDisposable 2020). 

In short, by reacting to the dehumanising effects of automated segregation 
by reaching for a ready-to-hand humanism, we are not escaping challenges 
like climate change and the politics of racial supremacy, or the underlying 
assumptions of human exceptionalism and subjectivity that are based on a 
dualistic metaphysics. 


New Materialist AI 


We are seeking an alternative AI that avoids the dehumanisation induced by 
automated segregation. Where AI is an engine of injustice it is because it inten- 
sifies the reductiveness, representationalism and universalism that privileges 
an existing social hegemony. At the same time, we recognise the dangers of a 
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reactive humanism; of fetishising human uniqueness in a way that perversely 
ensures some humans don’t make the cut, and whose bordering off of the rest 
of the material world reduces it to an exploitable resource. We’re looking for 
the possibility of post-AI that is at the same time postcolonial and posthuman 
(Mitchell 2015). 

However, this is not an exercise in reconciling theories. The aim is to sketch 
out a practice, or a praxis; an approach that can alter the current performativity 
of AI, not just critique it. So whatever we draw from the field of new material- 
ism (Sanzo 2018), from its fluidity of being and its immersive relationality, it 
needs to retain a clear possibility of agency. The aim is not simply to overcome 
dualism and reattach ourselves to a reality we have misunderstood, but to act 
politically against the amplification of injustice. 

The idea of a new materialist AI is important to explore because of the way 
it opens questions about the boundaries and hierarchies constructed between 
beings, and concerns itself with what these structures obscure and erase. The 
starting point here is the materiality of the world, but without any assumptions 
about meanings. The focus is on the way the material world and social mean- 
ings are part of a process of co-construction that is at the same time marked by 
relations of power. In other words, there’s a non-dualistic politics acting at the 
point of intersection between subjectivity and matter. 

AI takes sides here through its promotion of a worldview whose rigid catego- 
ries of meaning have real material consequences. The AI we know acts on and 
through individualised and itemised entities, and carries forward the political 
payload in terms of a world of atomised individuals and externalised nature. It 
reinforces particular boundaries in terms of what exists and how it gets distrib- 
uted. Seeking an alternative AI suggests it’s worth exploring a more posthuman 
approach, focusing on the interactions from which the familiar phenomena 
are emergent. Instead of an AI that takes the position of an outside observer, 
we can start with the idea of being as immersive and embedded, undermining 
the gaze of objectivity from which a single optimised truth can be affirmed. 

An immersive and emergent perspective on the world also suggests the 
idea of agency is no longer confined to the human but is distributed across 
the sociomaterial landscape. However, there are some drawbacks to distributed 
agency if we are attempting to construct a political project, especially the kind 
of distributed agency that falls under the umbrella of Actor-Network Theory. If 
the starting point is to describe complex networks of material-semiotic actors 
in a way that makes intentionality a secondary phenomenon, we open up the 
space for pacification; describing the becoming of what is, rather than striving 
for what should be. We are not simply seeking to reconnect to a non-dualistic 
reality, but to change it. 

One way to overcome dualism is by starting from the intertwining of phe- 
nomena that were previously classified as distinct. Our approach is to follow 
Karen Barad by identifying the way both the material world and subjects of 
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knowledge emerge through the actions of what she calls material-discursive 
apparatuses (Barad 2007). Her ideas of ‘agential realism bring together per- 
spectives from Foucault and from the quantum philosophy of Niels Bohr. 

The findings of quantum mechanics led Bohr to reject the assumption that 
the world is made of determinate objects with well-defined properties inde- 
pendent of specific experimental arrangements. Instead, phenomena are deter- 
mined by the wholeness of the measurement event. The particular way this is 
put together produces a particular division between the object and the observa- 
tion, which has the consequence of materialising some properties while exclud- 
ing others. This is an irrefutable experimental result at a quantum level but, as 
Barad spotted, parallels the way social constructivism analyses the formation 
of subjectivity through the operations of power. So she also draws on Foucault’s 
notion of a heterogeneous apparatus (‘dispositif’) of physical, administrative 
and knowledge structures that produces both us as social subjects and the soci- 
eties we inhabit. 

Barad uses the term ‘intra-action’ to talk about the mutual constitution of 
objects and subjects. Phenomena are produced by the intra-actions of appa- 
ratuses, which are active not passive; they are not just measuring instruments 
but boundary-drawing practices. These apparatuses are neither the labora- 
tory instruments of Bohr or the semiotic institutions of Foucault but both at 
the same time; they are ‘material-discursive’ apparatuses. 

Humans, according to Barad are part of the ongoing reconfiguration of the 
world produced by these apparatuses. ‘Humans (like other parts of nature) are 
of the world, not in the world, and surely not outside of it looking in. Humans 
are intra-actively (re)constituted as part of the world’s becoming’ (206). We 
have agency through our participation in the iterative production of reality 
and the space that exists within that for new possibilities. Human practices are 
‘agentive participants’ in the way phenomena are ‘sedimented out’ of this ongo- 
ing process. The idea of sedimentation makes us pay attention to the fact that 
the world we experience, and our experiencing of it, are not the starting point 
for analysis but are already the product of active processes. Material and mean- 
ing are not separate but the depositions from a dynamic that is not dualistic. 
Seen in this light, the systems of AI are aspects of a material-discursive appara- 
tus that are themselves sedimented out from other material-discursive systems, 
all of which are open to participatory reworking. 

Al as we currently know it is an instance of representation gone wrong, built 
on a foundation of the same mistake spread across the philosophical and politi- 
cal landscape. AI is not simply a layer of representation imposed on a solid 
ontology but part ofa stack of practices that splits subject and object all the way 
down. Its automated segregations are boundary-drawing practices that act in 
the world. We're not going to find a line of flight by means of a better mapping, 
a more accurate metaphysical correspondence. By throwing our hat in with 
agential realism we're also trying to switch allegiance to a process philosophy, 
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where the emphasis is not on being but on becoming. Instead of mirroring real- 
ity, it's about the making of realities; the important thing is to make meanings 
that matter. This approach to an alternative AI is pragmatic, based on the prin- 
ciple that knowing the world is inseparable from agency within it. Its a situated 
metaphysics that is committed to making a difference by not only overcoming 
dualisms but by overcoming the division between knowing and caring. 

Care exists in the shadow of the kind of detachment and abstraction that is 
valorised by AI. Care starts from a concern with exclusions and boundaries, 
and with the asymmetry of consequences for the most vulnerable (Bellacasa 
2017). Dematerialising the divides between observer and observed, subject and 
object, humanity and nature is an opening to a kind of caring cosmopolitics; 
the term Isabelle Stengers uses for being attentive and responsive to the multi- 
ples of being with which we are entangled and co-constituted (Mitchell 2015). 
This acceptance of heterogeneity without fixed boundaries and an interdepend- 
ence that is also an intra-dependence gives us the basis for an approach that 
is both posthuman and postcolonial. A situated caring means starting from 
the experiences of those at the margins, from ways of knowing that can chal- 
lenge the erasure of lived experience by the ideology of efficiency, in order to 
counter the algorithmic extension of carelessness. The question that remains is 
how we might go about applying ideas of agential realism and care to produce 
an alternative form of AI. 


Post-AI Politics 


Moving beyond the injustices powered by institutional machine learning 
means moving beyond representation to social recomposition. The material- 
discursive apparatus of AI acts to reinforce the wrong answer to the question of 
how to be together. It co-produces ineffective concepts of fairness and skewed 
distributions of social goods that reinforce each other and the status quo. As 
apparatuses that make meanings, the current instances of AI optimise Mark 
Fisher’s invocation of Frederic Jameson, that it is easier to imagine the end 
of the world than an end to capitalism (Fisher 2009) - it actively contributes 
to the former and erases the possibility of the latter. And yet we are agentive 
participants in this wider system whose intra-actions are open to reworking. 
We can be part of altering these boundary-drawing practices. The question of 
recomposition is the question of whether agential realism can be composed as 
collective action. 

Alis already earmarked as a solution to austerity through calculative rationing 
and optimised extensions of precarity and scarcity. At the time of writing, in the 
midst of the COVID-19 pandemic, the prospect of heightened post-pandemic 
austerity in an even more datafied and surveilled environment only boosts 
the likelihood of algorithmic optimisation being substituted for social justice. 
For many of the issues where Al is being applied to single out those deserving 
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intervention, that is, the most ‘risky; the situation would be made fairer overall 
if resources were redistributed to lower the overall risks. For example, rather 
than sinking resources into deep learning models that try to predict which 
parents will abuse their children, why not acknowledge that poverty and drug 
abuse are highly correlated with child abuse and put resources into reducing 
poverty and providing more accessible drug treatment services (Keddell 2015). 
Instead of seeing the problem as one of identifying the most risky ‘entity; it’s 
about starting from the inseparability of all entities and a recognition that they 
are co-constituting; not just an ethics of relationality (Birhane and Cummins 
2019) but an ethico-ontology of relationality. 

This approach doesn't depend on a top-down restructuring that moves from 
metaphysics to science to social policy. It simply requires an openness to the 
speculative starting point of agential participation motivated by care. It means 
acting ‘as if’ the intra-actions of a material-discursive apparatus could be 
determined by caring about the consequential meanings that are produced. In 
more familiar political terms, it means acting as if solidarity were not only a 
stance but a core facet of being, as if mutual aid was not simply a choice made 
after social reality was sedimented out but a driving element in the iterative 
reproduction of the world. What we are currently experiencing instead is not 
an established order but the entropic disorder established by apparatuses that 
utterly lack the balance necessary to sustain us or our world. 

The current and ongoing sedimentation of reality has its own pyroclastic 
momentum. What we can hope for at this time is to both slow it down and 
to diffract it through the introduction of difference. The pragmatic approach 
proposed here is to introduce structures that ‘slow the universalizing process 
by unsettling existing assumptions, boundaries and patterns of political action 
(Mitchell 2015). For this role we propose people’s councils for AI (McQuillan 
2018). People’s councils are bottom-up, federated structures that act as direct 
democratic assemblies. The mutual encounters and consensus-making of peo- 
ple’s councils are themselves transformative in terms of creating different rela- 
tionalities. The purpose of people’s councils is to become a mode of ‘presencing, 
of forcing the consideration of the unconsidered, or more fundamentally of 
reordering the idea of AI such that its production of pairings of concepts and 
material effects iterates towards an actually different society. 

People’s councils, based on solidarity and mutual aid, are an attempt to inocu- 
late our meaning-making structures against fascism. The operations of fascism 
past and present show the ability to embrace technology as technique while 
replacing modernism with a cult of authoritarian traditionalism, a disturbing 
tendency already visible in the ‘dark enlightenment’ narratives of neoreaction 
circulating in Silicon Valley (Haider 2017). AI as we know it forms a harmonic 
with neoreaction’s ‘near-sociopathic lack of emotional attachment’ and ‘pure 
incentive-based functionalism (MacDougald 2015). People’s councils are a 
diffraction of AI, introducing the difference of care as a mode of interference 
and superposition. 
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People’s councils are not a form of collectivised humanism or an attempt to 
re-centre the human as the only actor that matters, but a situated intervention 
in the ongoing reiteration of wider conditions. They are directed at the crea- 
tion of new possibilities. We still have the possibility of reforming the struc- 
tures, such as Al, that are increasingly becoming part of co-constituting us and 
our material world. The proposition is that these can be modes of differencing 
rather than of machinic modernism. The danger is that the mounting collat- 
eral damage caused by pervasive AI will drive a more atavistic response, whose 
boundary-drawing practices will increasingly be determined by notions such 
as racial superiority or eugenic justifications that some should be left to die in 
order to preserve the economy and/or the planet, narratives that we can already 
see emerging as a neoliberal and fascist reaction to the COVID-19 pandemic 
and as Malthusian responses to climate change. 

As a rule of thumb, we should examine every situation where AI is being 
offered as a solution and ask how on-the-ground collective action might enable 
a radical commoning of both risks and resources. Instead of a technocratic 
solution to precarious labour, for example, that imposes some spurious metric 
of fairness on a structure that embodies injustice, we look to a complete sociali- 
sation of the relations and materialities involved. This happens, for example, 
when workers react to layoffs by occupying their workplaces and transforming 
material production in collaboration with the local community (Pazos 2018). 
The only material-discursive politics consistent with a cosmopolitical care is a 
radical commoning. 

As Donna Haraway reminds us, our intra-actions and interdependencies 
stretch across vast fields of biota and abiota. Nevertheless ‘the doings of sit- 
uated, actual human beings matter. It matters with which ways of living and 
dying we cast our lot rather than others’ (Haraway 2016). Changes start with 
grassroots collectives who are prepared to take on the necessary activities of 
repair and resistance. The modelling which needs to take priority is not that 
delivered from on high by vast structures of computation but the modelling to 
each other of different ways of living and caring through mutual aid. Reclaim- 
ing political agency from engines of abstraction without the need for the rigid 
boundaries of humanism means taking solidarity as the starting point for 
our becoming. 
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PART 2 


Discourses and Myths 
About AI 


CHAPTER 6 


Ihe Language Labyrinth: Constructive 
Critique on the Terminology Used 
in the AI Discourse 


Rainer Rehak 


Introduction 


In the seventies of the last century, the British physicist and science fiction 
writer Arthur C. Clarke coined the phrase of any sufficiently advanced tech- 
nology being indistinguishable from magic - understood here as mystical 
forces not accessible to reason or science. In his stories Clarke often described 
technical artefacts such as anti-gravity engines, ‘flowing’ roads or tiny atom- 
constructing machinery. In some of his stories, nobody knows exactly how 
those technical objects work or how they have been constructed, they just use 
them and are happy doing so. 

In today’s specialised society with a division of labour, most people also do 
not understand most of the technology they use. However, this is not a serious 
problem, since for each technology there are specialists who understand, ana- 
lyse and improve the products in their field of work - unlike in Clarke's worlds. 
But since they are experts in few areas and human lifetime is limited, they are, 
of course, laypersons or maybe hobbyists in all other areas of technology. 
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After the first operational universal programmable digital computer - the 
Z3 - had been invented and built in 1941 in Berlin by Konrad Zuse, the rise 
of the digital computer towards today’s omnipresence started. In the 1960s, 
banks, insurances and large administrations began to use computers, police and 
intelligence agencies followed in the 1970s. Personal computers appeared 
and around that time newspapers wrote about the upcoming ‘electronic revo- 
lution’ in publishing. In the 1980s professional text work started to become 
digital and in the 1990s the internet was opened to the general public and to 
commercialisation. The phone system became digital, mobile internet became 
available and in the mid-2000s smartphones started to spread across the globe 
(Passig and Scholz 2015). 

During the advent of computers, they were solely operated by experts and 
used for specialised tasks such as batch calculations and book-keeping at large 
scale. Becoming smaller, cheaper, easier to use and more powerful over time, 
more and more use cases emerged up to the present situation of computer ubiq- 
uity. More applications, however, also meant more impact on personal lives, 
commercial activities and even societal change (Coy 1992). The broader and 
deeper the effects of widespread use of networked digital computers became, 
the more pressing political decisions about their development and regulation 
became as well. 

The situation today is characterised by non-experts constantly using comput- 
ers, sometimes not even aware of it, and non-experts making decisions about 
computer use in business, society and politics - from schools to solar power, 
from cryptography to cars. The only way to discuss highly complex computer 
systems and their implications is by analogies, simplifications and metaphors. 
However, condensing complex topics into understandable, discussable and 
then decidable bits is difficult in at least two ways. First, one has to deeply 
understand the subject and second, one has to understand its role and context 
in the discussion to focus on the relevant aspects. The first difficulty is to do 
with knowledge and lies in the classical technical expertise of specialists. But 
the second difficulty concerns what exactly should be explained in what way. 
Depending on the context of the discussion, certain aspects of the matter have 
to be explicated using explanations, metaphors and analogies highlighting the 
relevant technical characteristics and implications. Seen in this light, this prob- 
lem of metaphors for technology is not only philosophically highly interesting 
but also politically very relevant. Information technology systems are not used 
because of their actual technical properties, but because of their assumed func- 
tionality, whereas the discussion about the functionality is usually part of the 
political discourse itself (Morozov 2013). 

Given the complexity of current technology, only experts can understand 
such systems, yet only a small number of them actively and publicly take part 
in corrective political exchanges about technology. Especially in the field of 
artificial intelligence (AI) a wild jungle of problematic terms is in use. However, 
as long as discussions take place among AI specialists those terms function just 
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as domain-specific technical vocabulary and no harm is done. But domain- 
specific language often diffuses into other fields and then easily loses its con- 
text, its specificity and its limitations. In this process terms which might have 
started as pragmatic ‘weak’ metaphors within the technical field, then develop 
into proper technical terms eventually starting to be seen as ‘proper’ metaphors 
outside their original professional context. In addition to the effect of specific 
terms, those metaphors can also unfold effects beyond concrete technologies 
but also fuel or inhibit larger narratives around them or digital technology in 
general. Hence, powerful metaphors push the myths of unlimited potential of 
(computer) technology, the superiority of computation over human reasoning 
(Weizenbaum 1976) or the leading role of the ‘digital sublime’ in transforming 
society (Mosco 2004). On the other hand, less colourful and less visionary met- 
aphors keep such myths at bay and narratives grounded. Of course, it would be 
short-sighted to interpret the choice, development and dissemination of tech- 
nical metaphors and specifically AI terminologies purely as a somehow chaotic 
process of misunderstandings and unclear technical usage. Those discourses, as 
all discourses, are a playing field of interests and power where actors brawl over 
the ‘proper’ narration either because they find sincere truth in it (e.g., transhu- 
manist zealots of the singularity) or because it plainly benefits them politically 
or financially (e.g., companies selling AI), or both. Practically speaking, if rel- 
evant decision-makers are convinced that AI can develop a real ‘understanding’ 
or properly ‘interpret’ issues, its regular use for sensitive tasks like deciding 
about social benefits, guiding education, measuring behavioural compliance or 
judging court cases problematically looms; and corresponding companies will 
then eagerly come forward to sell such systems to them. 

All the above dynamics motivate this work to scrutinise the AI discourse 
regarding its language and specifically its metaphors. The paper analyses cen- 
tral notions of the AI debate, highlights their problematic consequences and 
contributes to the debate by proposing more fitting terminology and hereby 
enabling more fruitful debates. 


Conceptual Domains and Everyday Language 


Unlike the abstract field of mathematics, where most technical terms are eas- 
ily spotted as such, AI makes heavy use of anthropomorphisms. Considering 
Al-terms such as ‘recognition, ‘learning, ‘acting, ‘deciding, ‘remembering; 
‘understanding’ or even ‘intelligence’ itself, problems clearly loom across all 
possible conversations. Of course, many other sciences also use scientific terms 
that are derived from everyday language. In this case, these terms then have 
clearly defined meanings or at least linked discourses reflecting upon them. 
Examples are the terms ‘fear’ in psychology, ‘impulse’ in physics, ‘will’ in phi- 
losophy or ‘rejection’ in geology and ‘ideology’ mathematics. Often the same 
words have completely different meanings in different domains, sometimes 
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even contradictory meanings, as the examples of ‘work’ in physics and eco- 
nomic theory (energy transfer via application ofa force while moving an object 
vs. planned and purposeful activity of a person to produce goods or services) 
or ‘transparency’ in computer science and political science (invisibility vs. 
visibility) illustrate. 

Hence, problems arise when these scientific terms are transferred carelessly 
into other domains or back into everyday language used in political or public 
debates. This can occur through unprofessional science journalism, deliber- 
ate inaccuracy for PR purposes, exaggerations for raising third-party fund- 
ing, or generally due to a lack of sensitivity to the various levels and contexts 
of metaphors. 


The Case of Artificial Intelligence 


For some years now, technical solutions utilising artificial intelligence are 
widely seen as means to tackle many fundamental problems of mankind. From 
fighting the climate crisis, tackling the problems of ageing societies, reducing 
global poverty, stopping terror, detecting copyright infringements or curing 
cancer to improving evidence-based politics, improving predictive police work, 
local transportation, self-driving cars and even waste removal. 


Definitions 


The first step towards a meaningful discussion about AI would be to define 
what exactly one means when talking about AI. Historically there have been 
two major understandings of AI: strong AI or Artificial General Intelligence 
(AGI) and weak AI or Artificial Narrow Intelligence (ANI). The goal of AGI is 
the creation of an artificial human like intelligence, so an AI system with true 
human-like intelligence including perception, agency, consciousness, inten- 
tions and maybe even emotions (see Turing 1950 or more popular Kurzweil 
2005). ANI, on the other hand, refers to very domain-specific AI systems being 
able to accomplish very specific tasks in very narrowly defined contexts only. 
Questions of agency or consciousness do not arise with ANI systems, they are 
merely tools, although potentially very powerful tools. 

So far and tellingly, AGI can only be found in manifold media products 
within the fantasy or science fiction genre. Famous examples are Samantha in 
‘HER; Data in ‘Star Trek, HAL 9000 in ‘2001’ (based on a novel of the afore- 
mentioned writer Arthur C. Clarke), Bishop in Aliens, the Terminator in the 
movie series of the same name or even the Maschinenmensch in ‘Metropolis 
(Hermann 2020). 

In contrast, ANI systems are the ones calculating the moves in advanced 
chess games, the ones enhancing smartphone pictures, the ones doing pattern 
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recognition concerning speech (e.g., natural language processing) or images 
(e.g., computer vision) or even the ones optimising online search results. 
Furthermore, within the ANI discourse mainly two more specific definitions 
should be mentioned. The first one focuses on the technical process of how 
ANI works and goes along the lines of AI being computer algorithms that 
improve automatically through experience (cf. Mitchell 1997 about machine 
learning). The second one focuses more on the phenomenon of ANI by defin- 
ing AI broadly as computer systems that are able to perform tasks normally 
requiring human intelligence (Gevarter 1985). 

Technically there are a multitude of approaches to actually build AI systems. 
Those approaches are usually referred to as the field of machine learning (ML) 
and comprise the so-called symbolic approaches with explicit data representa- 
tions of relevant information like simple decision trees or formal logic-based 
ones like knowledge databases with inference algorithms. These approaches 
are comparatively limited due to the necessity of explicit data representation. 
Then again there are the more recent sub-symbolic approaches of ML which do 
not use explicit data representations of relevant information but mathemati- 
cal (e.g., statistical) methods for processing all kinds of data. Artificial neural 
networks (ANN) or evolutionary computation are examples of sub-symbolic 
approaches in ML. Interestingly so far, none of the actual methods available 
seem to promise a path to AGI. 

Yet, despite having at least some general definitions at hand, the common 
discussion usually ignores those and therefore the range of Al-assigned func- 
tionality reaches from applying traditional statistics to using machine learning 
(ML) techniques up to solely movie inspired ideas or even generally to ‘highly 
complex information systems, as in the official ‘Social Principles of Human- 
centric AT of Japan (Council for Social Principles of Human-centric AI 2019). 

In the following, we will concentrate on artificial neural networks to illus- 
trate the fallacies and pitfalls of questionably used language. The focus on ANN 
in this text is in line with the current debate of AI, where AI is predominantly 
used synonymously with machine learning using artificial neural networks 
(Eberl 2018). Nevertheless, the problems mentioned here also apply to debates 
concerning other forms of AI, when a similar terminology is being used. 

Key drivers for the current AI renaissance are the successes of applying arti- 
ficial neural networks to huge amounts of data now being available and using 
new powerful hardware. Although the theoretical foundations of the concepts 
used were conceived as early as the 1980s, the performance of such a system 
has improved to such an extent over the last years, that they can now be put to 
practical use in many new use cases, sometimes even in real-time applications 
such as image or speech recognition. Especially if huge data sets for training 
are available, depending on the task results can be much better than traditional 
symbolic approaches where information is written into databases for explicit 
knowledge representation. 


92 Al for Everyone? 


Before we analyse the language being used to describe the functionality, we 
should have a look at the inner workings of artificial neural networks to have a 
base for scrutinising terminologies. 


Basic Structure of Artificial Neural Networks 


Artificial neural networks are an approach of computer science to solve com- 
plex problems that are hard to explicitly formulate, or more concretely: to pro- 
gram. Those networks are inspired by the function of the human brain and 
its network of neurons; however, the model of a neuron being used is very 
simplistic. Many details of biological neuronal networks, such as myelination 
or ageing (Hartline 2009), are left out, as well as new mechanisms, such as back- 
propagation (Crick 1989), are introduced. Trying to follow the original model, 
each artificial neuron, the smallest unit of such systems, has several inputs 
and one output. In each artificial neuron, the inputs are weighted according to 
its configuration and then summed up. If the result exceeds a certain defined 
threshold the neuron is triggered, and a signal is passed on to the output. These 
neurons are usually formed into ‘layers, where each layer’s outputs are the next 
layer’s inputs. The resulting artificial neural network thus has as its input the 
individual inputs of the first layer, and as its output the individual outputs of 
the last layer. The layers in between are usually called ‘hidden’ layers and with 
many hidden layers an artificial neural network is usually called deep. 

In the practical example of image recognition, the input would consist of the 
colour values of all distinct pixels in a given image and the output would be 
the probability distribution among the predefined set of objects to recognise. 


Configuring the Networks 


From a computer science point of view ANNs are very simple algorithms, since 
the signal paths through the connections of the network can easily be calcu- 
lated by mathematical equations. After all, it is the variables of this equation 
(weights, thresholds, etc.) that accord the powerful functionality to ANNs. So 
ANNs are basically simple programs with a very complex configuration file and 
there are various ways of configuring artificial neural networks, which will now 
briefly be described. Building such a network involves certain degrees of free- 
dom and hence decisions, such as the number of artificial neurons, the number 
of layers, the number and weights of connections between artificial neurons and 
the specific function determining the trigger behaviour of each artificial neu- 
ron. To properly recognise certain patterns in the given data - objects, clusters 
etc. - all those parameters need to be adjusted to a use case. Usually there are 
best practices how to initially set it up; then the artificial network has to be fur- 
ther improved step by step. During this process the weights of the connections 
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will be adjusted slightly in each step, until the desired outcome is created, may 
it be the satisfactory detection of cats in pictures or the clustering of vast data in 
a useful way. Those training cycles are often done with a lot oflabelled data and 
then repeated until the weights do not change any more. Now it is a configured 
artificial neural network for the given task in the given domain. 


Speaking about the Networks 


Now we will take a closer look at how computer scientists speak about this tech- 
nology in papers and in public, and how those utterances are carried into jour- 
nalism and furthermore into politics. As mentioned above, the description of 
ANNs as being inspired by the human brain already implies an analogy which 
must be critically reflected upon. Commonly used ANNs are usually compara- 
tively simple, both in terms of how the biochemical properties of neurons are 
modelled but also in the complexity of the networks themselves. A comparison: 
the human brain consists of some 100 billion neurons while each is connected 
to 7,000 other neurons on average. ANNs on the other hand are in the mag- 
nitude of hundreds or thousands of neurons while each is connected to tens 
or hundreds of other neurons. This difference in orders of magnitude entails 
a huge difference in functionality, let alone understanding them as models of 
the human brain. Even if to this point the difference might only be a matter 
of scale and complexity, not principle, we have no indication of that changing 
anytime soon. Thus, using the notion of ‘human cognition’ to describe ANN 
is not only radically oversimplifying, it also opens up the metaphor space to 
other neighbouring yet misleading concepts. For example, scientists usually do 
not speak of networks being configured but being ‘trained’ or doing ‘(deep) 
learning. Along those lines are notions like ‘recognition, ‘acting; ‘discrimina- 
tion, ‘communication, ‘memory; ‘understanding’ and, of course, ‘intelligence’ 


Considering Human Related Concepts 


When we usually speak of ‘learning’ it is being used as a cognitive and social 
concept describing humans (or, to be inclusive, intelligent species in general) 
gaining knowledge as individual learner or as a group, involving other peers, 
motivations, intentions, teachers or coaches and a cultural background (Bieri 
2017). This concept includes the context and a whole range of learning pro- 
cesses being researched, tested and applied in the academic and practical fields 
of psychology of learning, pedagogy, educational science (Piaget 1944) and 
neuroscience (Kandel and Hawkins 1995). This is a substantial difference to 
the manual or automated configuration of an ANN using test sets of data. Seen 
in this light, the common notion of ‘self-learning systems’ sounds even more 
misplaced. This difference in understanding has great implications, since, for 
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example, an ANN would never get bored with its training data and therefore 
decide to learn something else or simply refuse to cooperate (Weizenbaum 
1976); metaphors matter, no Terminator from the movies in sight. 

‘Recognition’ or ‘memory’ are also very complex concepts in the human 
realm. Recognising objects or faces requires attention, focus, context and - 
depending on one’s school of thought - even consciousness or emotions. 
Human recognition is therefore completely different from automatically find- 
ing differences of brightness in pictures to determine the shape and class of 
an expected object (Goodman 1976). Furthermore, consciously remembering 
something is a highly complex process for humans which is more compara- 
ble to living through imagined events again and by that even changing what 
is being remembered. Human memory is therefore a very lively and dynamic 
process, and not at all comparable to retrieving accurate copies of stored data 
bits (Kandel and Hawkins 1995). 

Especially the notions of ‘action’ or even ‘agency’ are highly problematic 
when being applied to computers or robots. The move of a computer-controlled 
robotic arm in a factory should not be called a robots action, just because 
it would be an ‘action’ if the arm belonged to a human being. Concerning 
human actions, very broad and long-lasting discussions at least in philosophy 
and the social sciences already exist, note the difference between ‘behaviour’ and 
‘action’ (Searle 1992). The former only focuses on observable movement, 
whereas the latter also includes questions of intention, meaning, conscious- 
ness, teleology, world modelling, emotions, context, culture and much more 
(Weizenbaum 1976). While a robot or a robotic arm can be described in terms 
of behavioural observations, its movements should not easily be called actions 
(Fischer and Ravizza 1998). 

Similarly complex is the notion of communication’ in a human context, since 
communication surely differs from simply uttering sounds or writing shapes. 
‘Communication requires a communication partner, who knows that the sym- 
bols used have been chosen explicitly with the understanding that they will be 
interpreted as deliberate utterances (von Savigny 1983). Communication there- 
fore needs at least the common acknowledgement of the communicational pro- 
cess by the involved parties, in other words an understanding of each other as 
communicating (Watzlawick 1964). A ‘successful’ communication is then the 
result of both parties agreeing that it was successful and therefore the crea- 
tion of a common understanding. Hence, the sound of a loudspeaker or the 
text on a screen does not constitute a process of communication in the human 
sense, even if their consequences are the production of information within the 
receiving human being. If there is no reflection of the communication partner, 
no deliberation, no freedom of which symbols to choose and what to commu- 
nicate one should not easily apply such complex notions as ‘communication 
outside its scope without explanation. 

Furthermore, the concept of ‘autonomy’ - as opposed to heteronomy or 
being externally controlled - is widely used nowadays when dealing with 
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artificial intelligence, may it be concerning ‘intelligent’ cars or advanced’ weapon 
systems. Although starting in the last century even human autonomy has been 
largely criticised within the social sciences (some even say completely decon- 
structed, Krahnke 2006) since individuals are largely influenced by culture, 
societal norms and the like, the concept of autonomy seems to gain new trac- 
tion in the context of computer science. Yet, it is a very simplistic understanding 
of the original concept (Gerhardt 2002). Systems claimed to be ‘autonomous’ 
heavily depend on many factors, e.g., a stable, calculable environment, but also 
on programming, tuning, training, repairing, refuelling and debugging, which 
are still traditionally done by humans, often with the help of other technical 
systems. In effect those systems act according to inputs and surroundings, but 
they do not ‘decide’ on something (Kreowski 2018), certainly not as humans do 
(Bieri 2001). Here again, the system can in principle not contemplate its actions 
and finally reach the conclusion to stop operating or change its programmed 
objectives autonomously. Hence, artificial intelligence systems - with or with- 
out ANN - might be highly complex systems, but they are neither autonomous 
nor should responsibility or accountability be attributed to them (Fischer and 
Ravizza 1998). Here we see one concrete instance of the importance of differ- 
entiating between the domain-specific ANI and universal AGI (Rispens 2005). 
This clarification is not meant to diminish the technical work of all engineers 
involved in such ‘autonomous’ systems, it is purely a critique about how to ade- 
quately contextualise and talk about such systems and its capabilities in non- 
expert contexts. 


Instances and Consequences 


After having briefly touched upon some areas of wrongly used concepts, we can 
take a look at concrete examples, where such language use specifically matters. 

A very interesting and at that time widely discussed example was Google's 
‘Deep Dream’ image recognition and classification software from 2015, code- 
name ‘Inception. As described above, ANNs do not contain any kind of explicit 
models; they implicitly have the ‘trained’ properties distributed within their 
structures. Some of those structures can be visualised by inserting random data - 
called ‘noise’ - instead of actual pictures. In this noise, the ANN then detects 
patterns exposing its own inner structure. What is interesting are not the 
results - predominantly psychedelic imagery - but the terminology being used 
in Google’s descriptions and journalists’ reports. The name ‘Deep Dream’ alone 
is already significant, but also the descriptive phrases ‘Inside an artificial brain’ 
and Inceptionism (Mordvintsev and Tyka 2015). Both phrases (deliberately) 
give free rein to one’s imagination. In additional texts provided by Google, 
wordings such as ‘the network makes decisions’ accumulate. Further claims are 
that it ‘searches’ for the right qualities in pictures, it ‘learns like a child’ or it 
even ‘interprets’ pictures. Using this misleading vocabulary to describe ANNs 
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and similar technical artefacts, one can easily start to hope that they will be 
able to learn something about the fundamentals of human thinking. Presum- 
ably those texts and descriptions have been written for the primary purpose 
of marketing or public relations, since they explain little but signify the abili- 
ties and knowledge of the makers, yet that does not diminish the effect of the 
language used. For many journalists and executive summary writers or even 
the interested public those texts are the main source of information, not the 
hopefully neutral scientific papers. In effect, many of those misleading terms 
where widely used, expanded on and by that spread right into politician’s daily 
briefings, think-tank working papers and dozens of management magazines, 
where the readers are usually not aware of the initial meanings. This distorted 
‘knowledge’ then becomes the basis for impactful political, societal and mana- 
gerial decisions. 

Other instances where using wrong concepts and wordings mattered greatly 
are in car crashes involving automated vehicles, e.g., from companies like Uber, 
Google or Tesla. For example, in 2018 a Tesla vehicle drove into a parked police 
car in California, because the driver had activated the ‘autopilot’ feature and did 
not pay attention to the road. This crash severely exposed the misnomer. The 
driver could have read the detailed ‘autopilot’ manual before invoking such a 
potentially dangerous feature, yet, ifthis mode of driving had been called ‘assisted 
driving’ instead of ‘autopilot, very few people would have expected the car to 
autonomously drive ‘by itself: So, thinking about a car having an autopilot is 
quite different from thinking about a car having a functionality its makers call 
‘autopilot. Actually reading into Teslas manuals, different levels of driving 
assistance are being worked on, e.g., Enhanced Autopilot’ or ‘Full Self-Driving, 
whereas the latter has not been implemented so far. Further dissecting the 
existing ‘autopilot’ feature one finds it comprises different sub-functionalities 
such as Lane Assist, Collision Avoidance Assist, Speed Assist, Auto High Beam, 
Traffic Aware Cruise Control or Assisted Lane Changes. This collection of assis- 
tance technologies sounds very helpful, yet it does not seem to add up to the 
proclaimed new level of autonomous driving systems with an autopilot being 
able to ‘independently’ drive by itself. 

Those examples clearly show how a distinct reality is created by talking about 
technology in certain terms, yet avoiding others. Choosing the right terms, is 
not always a matter of life and death, but they certainly pre-structure social and 
societal negotiations regarding the use of technology. 


Malicious Metaphors and Transhumanism 


Suddenly we arrive in a situation where metaphors are not only better or worse 
for explaining specifics of technology, but where specific metaphors are delib- 
erately being used to push certain agendas; in Tesla’s case to push a commer- 
cial and futurist agenda. Commercial because of using ‘autonomy’ as a unique 
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selling point for cars and futuristic, as it implies that ‘autonomy’ is a neces- 
sary and objective improvement for everyone’ life and the society as a whole. 
Generally, most innovative products involving ‘artificial intelligence’ and ‘next 
generation technology’ are being communicated as making ‘the world a better 
place, ‘humans more empowered’ or ‘societies more free’ by the PR depart- 
ments of the offering companies and spread even further by willing believers 
and reporting journalists. The long-standing effects of metaphors let loose can 
also be seen vividly in the discourse about transhumanism, where humans 
themselves, even humankind as a whole, should be enhanced and improved 
using (information) technology, predominantly by using AI. Here again the 
proponents either really believe in or profit from those narratives, or both 
(Kurzweil 2005). 

In this discourse all mistakes of the AI terminology can be observed fully 
developed with many consequences, since when we pose the transhumanist 
question regarding how information technology can help human beings the 
answer is usually ‘enhancement’. Yet the notion of ‘enhancement’ is being used 
in a very technical way, ignoring its fundamental multiplicity of meanings. 
With information technology, so the argument from the classic flavour of trans- 
humanism goes, we will soon be able to fix and update the human operating 
system: merging with intelligent technical systems will make our brain remem- 
ber more faces, forget less details, think faster, jump higher, live longer, see 
more sharply, be awake longer, be stronger, hear more frequencies and even 
create new senses - exactly how a technologist would imagine what new tech- 
nology could deliver for humanity (Kurzweil 2005). More recent concepts see 
humans and AI systems in a cooperative even symbiotic relationship. Those 
concepts exemplify the direction of imagination once we assume there are truly 
‘intelligent’ systems with ‘agency’ who can ‘decide’ and ‘act. 

However interestingly the underlying and implicit assumption is a very 
specific - to be precise: technical - understanding of what is considered ‘good’ 
or desirable: But does every human or even the majority primarily want to 
remember more, forget less, live longer or run faster? Are those aspects even 
the most pressing issues we want technology to solve? In addition, not only do 
those fantasies happily follow along the lines of the neo-liberal logic of apply- 
ing quantification, competition, performance and efficiency into all aspects of 
life, they also unconsciously mix in masculinist - even militarist - fantasies 
of power, control, strength and subjugation of the natural or finally correcting 
the assumed defective (Schmitz and Schinzel 2004). 

As valid as those opinions concerning optimisations are, still it is impor- 
tant that views like that imply absolute values and are incompatible with views 
which put social negotiation, non-mechanistic cultural dynamics or in general 
pluralistic approaches in their centre. To structure the discourse, I call those 
conflicting groups of views regimes of enhancement. Clearly it is not possible to 
‘enhance’ a human being with technically actualised immortality, if this person 
does not want to live forever or does not find it particularly relevant. Many 


98 Al for Everyone? 


other conflicting views can be thought of. However, the mere acceptance of the 
concept of regimes already breaks any claim for absoluteness and opens 
the door for discussing different understandings of ‘enhancements. Accepting 
this already makes any positions somehow compatible and allows for indivi- 
dual or even societal endeavours of creative re-interpretations of the concept 
of transhumanism itself (Haraway 1991). 

The transhumanist discourse outlines the consequences of not reflecting on 
core notions like ‘enhancement’ in the same way as it is consequential not to 
reflect on ‘intelligence’ in the AI discourse (Bonsiepen 1994). Broken visions 
and faulty applications are to be expected. Furthermore, this kind of language 
also shapes and attracts a certain kind of mindset where the above mentioned 
reductionist metaphors are not even used as metaphors anymore, but as accu- 
rate descriptions of the world (Coy 1993). 


Constructive Wording 


So, next time decision-makers and journalists will be asked about possibili- 
ties of technology they will surely remember having heard and read about 
computers winning Chess and Go, driving cars, recognising speech, translat- 
ing text, managing traffic and generally finding optimal solutions to given 
problems (Dreyfus 1972). But using deficient anthropomorphisms like 
‘self-learning, ‘autonomous or ‘intelligent’ to describe the technical options 
of solving problems will lead to malicious decisions (The Royal Society 
2018, 7-8). 

Surely the best solution for this problem would be to completely change the 
terminology, but since large parts of the above mentioned are fixed scientific 
terms, a clean slate approach seems unrealistic. Therefore, at least in interdisci- 
plinary work, science journalism activities or political hearings, a focus should 
be put on choosing the appropriate wording by scientists and (science) jour- 
nalists. Only then policy and decision-makers have a chance to meaningfully 
grasp the consequences of their actions. In addition, interdisciplinary research 
could also get a more solid (communication) ground. Of course, this change of 
terms will not be the end of discipline-limited jargon in AI, but it would surely 
increase the efficiency of exchange between the different fields. 

For concretely deciding which terms to use and which words to change it 
would be generally preferable to have some kind of criteria. Following the 
above descriptions, the used terminology should be as close as possible to 
the technical actualities while at the same time avoiding: 


e technical terms that have a connotation in common language reaching far 
beyond the actual technical function, e.g., recognition, agent, communica- 
tion, language, memory, training, senses, etc. since they will be understood 
as metaphors 


The Language Labyrinth 99 


e anthropomorphisms which are not technical terms but usually used as 
metaphors to describe technical details, e.g., thinking, (re)acting, deciding, 
remembering, etc. 

e concepts widely used in popular science, media and science fiction imply- 
ing a completely different meaning e.g., intelligent machine, android, self- 
improving, autopilot, etc. 


Certainly those words can be replaced by more fitting vocabulary. Depend- 
ing on context ‘remembering’ could be paraphrased by ‘implicitly stored in 
configuration, ‘learning’ by ‘changing/improving configuration, ‘recognition 
by ‘detection, ‘intelligent’ by ‘automated’ (cf. Butollo 2018), ‘action’ by ‘move- 
ment’ or ‘response; ‘decision’ or judgement’ by ‘calculation (cf. Weizenbaum 
1976), and ‘communication’ by ‘indication’ or ‘signalling: However, terms like 
‘agency’ and ‘autonomy’ should be discarded entirely, since they are neither 
accurate or necessary nor helpful; they are just completely misleading. 

Being aware that this change might also bear consequences for scientific grant 
proposals which usually have to sound societally important, scientifically inno- 
vative and relevant, it is imperative here too as part of science ethics to reflect 
on the wider consequences of the language used to communicate. Admittedly 
it should be noted that those suggestions won't be applied by speakers who are 
deeply convinced of such metaphors fitting the subject matter, yet, they would 
then be clearly visible as such. 


Closing Remarks 


Technology is used and politically decided upon perceived functionality, not 
upon the actually implemented functionality. However, communicating func- 
tionality is much more driven by interests than creating the actual technology. 
Therefore, attribution ascription is a very delicate and consequential issue that 
paints a differentiated picture of the consequences of careless use of terms. If 
relevant decision-makers in politics and society are (really) convinced at some 
point that these ‘new’ artificial neural networks can develop an understanding 
of things or properly interpret facts, nothing would stand in the way of their 
use for socially or politically sensitive tasks like deciding about social benefits, 
teaching children or judging court cases. Here the difference between ‘judging’ 
and judging, ‘acting’ and acting play out. If one acts in the social science mean- 
ing of the word, one has to take responsibility for one’s actions, if a computer 
only ‘acts, used as a metaphor, responsibility is blurred. 

Hence, especially computer professionals but also scientific journalists should 
follow the professional responsibility to be more sensitive about the criticised 
misleading metaphors and in effect to change them to more fitting ones. The 
danger here does not lie in incompletely understanding computers or AI but 
in not understanding them while thinking that they have been understood. A 
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possible way out of this tricky situation is certainly more disciplinary openness 
towards interdisciplinary research and communication. Especially the disci- 
pline of computer science could embrace this kind of exchange much more, 
from student curricula to research projects. This would maybe not so much 
change their disciplinary core work but it would contextualise this work, cre- 
ate better accessibility for other less technical fields and produce overall more 
useful results. Naturally, this would require all parties involved to speak to each 
other but also to listen and teach each other ways of looking at the world. Of 
course, and not least those ventures must be encouraged and facilitated by field 
leaders, research grant givers and research politics alike. 

So, if technological discussions and societal reflections on the use of techno- 
logy are to be fruitful, scientists and (science) journalists alike have to stop join- 
ing the buzzword-driven language game of commercial actors and AI believers 
alike, which does neither help with solutions nor advances science. It merely 
entertains our wishful thinking of how magical technology should shape the 
future. Finally, we record that a chess computer will never get up and change its 
profession, exponential growth in computing power does so far not entail more 
than linear growth of cognitive-like functionality, and the fear of computers 
eliminating all human jobs is a myth capable of inciting fear since at least 1972. 

But maybe, indeed, any sufficiently advanced technology is indistinguishable 
from magic - to the layperson - but we also have to conclude that this ‘magic’ 
is being constructed and used by certain expert ‘magicians’ to advance their 
own interests and agendas, or that of their masters (Hermann 2020). So not 
even such magical interpretation would spare us the necessity to pay attention 
to power, details and debate (Kitchin 2017). This chapter tries to constructively 
be a part of this interdisciplinary project. 
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CHAPTER 7 


AI Ethics Needs Good Data 


Angela Daly, S. Kate Devitt and Monique Mann 


Introduction 


Artificial Intelligence (AI) is increasingly a part of our societies and econo- 
mies, principally paid for and benefiting private organisations and govern- 
ments. Al applications are offered free to consumers and end-users in products 
such as Google Maps in exchange for access to vast amounts of data (Zuboff 
2019). While AI has the potential to be used for many socially beneficial pur- 
poses, there is concern about dangerous and problematic uses of the technol- 
ogy, which has prompted a global conversation on the normative principles 
to which AI ought adhere, under the banner of ‘AI ethics. Governments, cor- 
porations and NGOs throughout the world have generated their own sets of 
Al ethics principles. Questions and critiques arise about the content of these 
ethics principles, whether they are actually implemented, and their (legal) 
enforceability (Wagner 2018). Broader issues emerge about the power and 
privilege of the organisations, governments and individuals which are creat- 
ing and implementing AI and accompanying ethical principles. For example, 
Google has recently announced an ethics service (Simonite 2020), yet has been 
mired in ethics controversies from violating privacy law (Finley 2019), work- 
ing on controversial military projects (Crofts and van Rijswijk 2020) and dis- 
solving its Ethics Board merely a week after its establishment (Statt 2019). The 
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creation of, and compliance with, ethical frameworks can be expensive in time 
and resources, making it easier for wealthy organisations and nation-states to 
comply with ethical governance and even profit from it while maintaining fun- 
damentally inequitable, unjust and self-protecting practices. 

In this chapter, we argue that we need to focus more on the broader question 
of power and privilege than merely the aforementioned and depoliticised lan- 
guage of AI ethics. Al ethics principles and frameworks tend to centre around 
the same values (fairness, accountability, transparency, privacy, etc.) and are 
insufficient to address the justice challenges presented by AI in society. Indeed, 
Amoore (2020, 81) contends that ‘a cloud ethics must be capable of asking 
questions and making political claims that are not already recognised on the 
existing terrain of rights to privacy and freedoms of association and assembly 
This can be connected to arguments made by Hoffman (2019, 907) that a focus 
on a ‘narrow set of goods, namely rights, opportunities, and resources’ is limit- 
ing in that it ‘cannot account for justice issues related to the design of social, 
economic, and physical institutions that structure decision-making power and 
shape normative standards of identity and behaviour’. Hoffman (2019, 908) 
also contends that an ‘outsized focus on these goods obscures dimensions of 
justice not easily reconciled with a rubric of rights, opportunities and wealth 
and that ‘emphasis on distributions of these goods fails to appropriately attend 
to the legitimating, discursive, or dignitary dimensions of data and information 
in its social and political context’. 

In light of these nascent critiques, we presenta politically progressive approach 
to AI governance based on ‘good data’ which seeks to empower communities 
and progress the priorities of marginalised and disenfranchised groups world- 
wide. Our approach also moves the conversation beyond anthropocentrism by 
incorporating ATs environmental impact into normative discussions (see Foth 
et al. 2020). Data is the fuel for AI, providing value and power. AI capabilities 
are typically designed, funded, developed, deployed and regulated (if indeed at 
all) by the wealthy progressing the values of profit, power and dominance. AI 
is constructed in a way that typically reinforces and cements the status quo and 
existing power relationships. AI will continue to be unethical without political 
consciousness regarding the actors and scenarios into which it is being con- 
ceptualised, designed and implemented and the actors and scenarios that are 
currently excluded from consideration. Our Good Data approach instead seeks 
to bring these actors, issues and scenarios clearly into the spotlight and thereby 
into the normative conversation on AI and digital technology more generally. 

Accordingly, the chapter will offer an overview and critique of AI ethics, 
before presenting a conceptual analysis of Good Data in the context of AI. We 
advance Good Data as an alternative framing for ethical (in the broad sense) 
questions involving digital data and conclude with some directions on how 
Good Data can be implemented in practice vis-a-vis AI. However, for a ‘Best 
Data’ scenario for AI to be achieved, greater change contesting and replacing 
neoliberal capitalism may be necessary (Daly 2016; Zuboff 2019), given the 
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political economy roots of many contemporary Bad Data practices by gov- 
ernments and large corporations throughout the world, which are also being 
implemented via AI applications. Thereby any ‘quick fixes’ offered by AI eth- 
ics principles may be illusory and indeed longer term more comprehensive 
approach/es to goodness in AI, data and society overall are needed. 


AI Ethics and Governance 


In the last few years, a global debate and discussion has emerged about govern- 
ing Al and, in particular, whether and to which norms AI should adhere. This 
debate acknowledges the possibility and actuality of AI being used for nor- 
matively problematic purposes, including in physically dangerous and other 
harmful ways, as well as what ethical approaches humans should take towards 
potentially autonomous AI that may mimic our own characteristics (see e.g., 
Bennett and Daly 2020; Donath 2020; Dörfler 2020). A variety of stakeholders, 
from nation-states throughout the world to regional blocs like the European 
Union to large technology companies (both US- and China-based) to religious 
institutions have participated in this debate by issuing their own iterations of 
‘ethics’ principles to which AI ought adhere (Daly et al. 2019). There is now 
also a corresponding blossoming of academic literature on AI ethics from a 
number of disciplines from computer science to law to philosophy to engineer- 
ing examining, collating, comparing and critiquing these ethics statements and 
proposals (see e.g., Fjeld et al. 2020; Larsson 2020). 

There have been two prominent critiques of this ‘turn to ethics’: one related 
to the form of these ethics initiatives and one relating to the substance. 

One prominent critique of this ‘turn to ethics’ has been from Wagner (2018), 
who has expressed concerns about these initiatives constituting ‘ethics wash- 
ing’ or ‘ethics shopping’ - that ‘ethics’ may ‘provide an easy alternative to gov- 
ernment regulation, in the context of strong regulation (especially containing 
fundamental rights protections) having encountered resistance from industry 
players. The majority of the ethics statements to date, even those from nation- 
states and other public actors, do not have legally binding force. This raises the 
question of how sincere and effective such principles may be, particularly in 
the context of a world in which large technology companies are very power- 
ful, have engaged in problematic conduct in the past and at present, and are 
not always well regulated by governments (Daly 2016). It is these pre-existing 
‘infrastructures’ and scenarios into which Al is being developed and deployed, 
yet these aspects are often divorced from the AI ethics discourses (Veale 2020). 
In addition, we see the involvement of industry players in defining these ethical 
principles, with the EU High-Level Expert Group a notable and controversial 
example as the presence of industry lobbying seemingly had an impact on the 
final document (see e.g., Rességuier and Rodrigues 2020). Law is one way of 
enforcing ethical principles but not the only one, and Hagendorff (2020) points 
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to broader issues with AI ethics ‘usually lack[ing] mechanisms to reinforce its 
own normative claims, mechanisms that include law but also cover technical 
implementations and business process operationalisations of ‘ethics. 

It is also important to note the ‘weaponisatior of ‘ethics’ in the AI debates, 
to refer to these ethics initiatives issued mainly by governments and corpora- 
tions and the ‘ethics-washing’ critiques which can descend into ‘ethics bash- 
ing, distracting from the more general and broad meaning of ethics as it relates 
to morality and virtue (Bietti 2020). In other words, ‘ethics’ has been used in a 
specific way in the AI debates both to promote (usually non-binding) lists of 
norms and as a target for critique that these norms are insufficient, formulated 
by the wrong actors and not backed up with enforcement or implementation. 
However, ‘ethics’ in the more general term can make an important contribu- 
tion to considerations of morality in a broad sense when it comes to AI (Bietti 
2020): the ethics baby should not be thrown out with the ‘AI ethics initiatives’ 
bathwater. 

The critique of ethics initiatives’ substance relates to what is included and 
excluded as normative principles. Hagendorff (2020) identifies recurring 
norms in ethics declarations he analyses and compares, notably ‘accountability, 
privacy or fairness: Along with ‘robustness or safety’ Hagendorff (2020) consid- 
ers these frequently occurring norms as those that ‘are most easily operation- 
alised mathematically and thus tend to be implemented in terms of technical 
solutions. Hagendorff (2020) also points to the ‘omissions’ from many AI ethics 
frameworks comprising ‘red lines’ on uses of AI which should be prohibited, 
political abuse of AI systems and the “hidden” social and ecological costs’ of 
AI systems. Furthermore, ethical AI principles such as fairness, accountability, 
transparency judge only the information systems within which Al is installed, 
without stepping back to analyse the socio-economic and political realities of 
the organisations which own and use the data - as per Bigo et al’s (2019) data 
politics - and AI, and the people who willingly or ignorantly provide the fuel 
to power AI. 

The critique of AI ethics’ substance also extends to scenarios where AI eth- 
ics are backed by legal enforceability. While a latecomer to state-led AI ethics, 
the United States has made up for lost time since 2019 starting with the Trump 
Administration Executive Order on Maintaining American Leadership in Arti- 
ficial Intelligence (US White House 2019). What is notable about this Executive 
Order is its legal enforceability albeit in the form of a direction to other US gov- 
ernment agencies rather than, for example, a piece of general legislation con- 
taining a set of legally binding normative principles (Daly et al. 2020). The Exec- 
utive Order does contain five high level principles, including the US driving the 
development of appropriate technical standards and protecting civil liberties and 
privacy (US White House 2019). However, the US’s policy among its own 
government agencies since then has promoted a deregulatory approach to AI 
whereby agencies should reduce regulatory barriers to AI development and 
adoption (Daly et al. 2020). 
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This approach demonstrates the limits of legal enforceability in AI eth- 
ics contexts whereby, in the US case, legal enforceability is used to mandate a 
deregulatory approach (Daly et al. 2020). Accordingly, ‘the legal enforceability 
of AI governance and ethics strategies does not necessarily equate to substan- 
tively better outcomes as regards actual AI governance and regulation (Daly 
et al. 2020). Instead, we must be alert to legal enforceability as a form of ‘law 
washing’ or when the binding force of law does not in itself prevent unethical 
uses of AI (Daly et al. 2020). 

This leads to the need to interrogate and evaluate both the form and sub- 
stance of AI ethics - what they do and do not contain in substance, and what 
binding force they may or may not have, whether they are only ‘performative’ 
and ‘instrumentalist’ or whether the language of ethics is only performative. In 
other words, AI ethics need a Good Data approach. 


A Good Data Approach 


Ethics as currently utilised in the AI debates is a limited frame through which 
Al issues can be viewed. While we acknowledge that ethics has a broader and 
more general sense than its use in AI ethics so far (Bietti 2020), we do not seek 
to reclaim it as a linguistic device given the term’s history and tarnishment in 
these debates. Instead, we propose ‘Good Data (Daly, Devitt and Mann 2019), 
as a more expansive concept to elucidate the values, rights and interests at stake 
when it comes to ATs development and deployment as well as that of other 
digital technologies. In particular, we argue that discourses, design and deploy- 
ment on and of AI must engage with power and political economy, perspectives 
which are largely lacking in AI ethics initiatives to date (see Johnson 2019). 

We conceived the notion of ‘Good Data’ to move beyond critique of the digi- 
tal (in which we have participated and continue to do so) to the (re)imagining 
and articulating of a more optimistic vision of the datafied future and, in par- 
ticular, how digital technologies and data, including but not limited to AI, can 
be used to further wider social, economic, cultural and political goals (Daly, 
Devitt and Mann 2019). We draw on work on data justice (Dencik, Hintz and 
Cable 2016), data activism (Milan and van der Velden 2016) and data politics 
(Bigo, Isen and Ruppert (2019) as key elements or examples of Good Data, 
while our concept involves ‘broader visions of goodness or ethics or politically 
progressive data’ (Daly, Devitt and Mann, 2019). 

AI ethics frameworks to date focus on evaluating the design and impacts of 
AI without sufficient attention on the socio-political contexts in which AI is 
developed and employed (as per Amoore 2020). What this means is that AI 
can be responsible, governable, trusted, equitable, traceable to the persons to 
whom it applies and reliable within the complex systems it is deployed - i.e., 
‘ethical’ by the measure of many of the AI ethics initiatives - but the govern- 
ing organisation/s responsible for the Al itself may be unethical in the broader 
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society and environment within which it is deployed. A Good Data approach 
interrogates these broader situations and factors which are often absent from 
AI ethics initiatives. 

It is reasonable to consider the scope of ‘goodness’ and what philosophical 
commitments we might have to the assertion of ‘good’ data. For our purposes, 
goodness can be a property of a thing, a service, a method, an event, a system, 
a process, a judgement, a sensation, a feeling or a combination of these. To 
identify ‘good, we could suppose that there are moral facts (see Parfit 2011; 
Scanlon 2014). Agreement on moral facts enables standards, policies, practices 
and frameworks to improve information systems and communicate expecta- 
tions. However, given the limits of our knowledge of moral facts (should they 
exist) and in light of colonial and post-colonial data practices (Arora 2016) we 
assume a hybrid moral theory - where we allow that some moral facts may be 
objective (e.g. ‘tolerance’ or ‘openness’) and others relative (e.g. Wong 1984). A 
hybrid theory allows respect for cultural diversity and demands case-by-case 
determinations of goodness and systematic values and standards. By promot- 
ing a hybrid account, we are prepared for disagreement about what is good and 
assume that the discovery of moral facts (if they do exist) is non-trivial 
and unresolved. We advocate an ethic of active seeking, openness and tolerance 
to diverse views on ‘the good’ particularly, and perhaps stridently, consultation 
with the underrepresented, marginalised and unheard. 


Pillars of Good Data 


Good Data contribute to understanding and justifying progressive political 
action by collectives. Good Data is thus situated in an ethical perspective to 
progress society, rather than simply satisfying an epistemic goal to inform. 
Therefore, we connect Good Data with political action and social justice - it 
means doing something good in the world, or equally not doing something 
bad, i.e. forbearance. Good Data also can take place and be relevant to all 
stages in the data collection process, from the beginning to the end encom- 
passing: when the decision is made for the data to be collected for AI use and 
by whom; at the point the data is collected by AJ; at the point the data is pro- 
cessed/analysed by AI; at the point the data is used by AI; and at the point 
the data is reused by AI. In order to conceptualise the process and outcome 
of Good Data, we advance these ‘pillars’ on which it should rest, rather than 
principles to which it should apply. 

We present four pillars: Community, Rights, Useability, Politics that emerge 
from the corpus of Good Data (Daly, Devitt and Mann 2019) which can guide 
digital technologies and data development, including for AI. We propose that 
these pillars can guide an ethical and politically progressive approach to AI 
development, governance and implementation. 
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Community 


Good Data must be orchestrated and mediated by and for individuals and col- 
lectives. Individuals and collectives should have access to, control over and 
opportunities to use their own data to promote sustainable, communal liv- 
ing, including communal sharing for community decision-making and self- 
governance and self-determination (see Lovett et al. 2019; Ho and Chuang 
2019). Data collection, analysis and use must be orchestrated and mediated by 
and for data subjects and communities advancing their data and technological 
sovereignty, rather than determined by those in power (Kuch et al. 2019; Mann 
et al. 2020). AI, constructed by communities, should be designed to assist com- 
munity participation in data-related decision-making and governance. This 
community element is usually absent from AI ethics initiatives, both in their 
formation - principally by elites - and their content. Commitment to theories 
and practices of sovereignty (Kurtulus 2004) are critical to systems’ architecture 
design including data permissions, accessibility and privacy. 


Rights 


While we recognise that the discourse of rights is limiting (as per Amoore 2020; 
Hoffman 2019), Good Data should still be collected with respect to humans 
and their rights, and that of the natural world, including animals and plants 
(Trenham and Steer 2019). The rise of big data and AI makes individual control 
over all their shared digital personal or community data a possibly insurmount- 
able task. Rights language and power stems from a protection of the individual 
(especially in western worldviews), and there may be conflicts between the 
community's values and priorities with community data and the preferences 
of an individual within those communities. While Kalulé and Joque (2019) 
criticise the contemporary anthropocentrism of AI and privileging of the west- 
ern human, we believe that a rights discourse is not completely futile and in 
principle AI can be developed to improve abidance with human rights and the 
rights of the environment. However, such a language of rights and especially 
the rights of the non-human are usually absent from AI ethics initiatives. With 
Good Data we urge that the environmental cost and impact of AI technolo- 
gies is an ‘externality which must be ‘internalised’ in discussions of ethics and 
politically progressive AI and digital data. 


Usability 


Good Data is usable and fit for purpose, consensual, fair and transparent 
(Trenham and Steer 2019). Measures of fairness and other values attrib- 
uted to data should be explicit (McNamara et al. 2019), and extend beyond 
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narrowly conceived technical explanations to challenge broader structural/ 
societal unfairness (see Hoffmann 2019 on the limits of fairness’). Data 
driven technologies must respect interpersonal relationships such as appro- 
priate (Flintham et al. 2019), e.g. members of the same household may wish 
for limits on accessing each other’s data - in other words, data is relational. 
Good Data is dependent on context, and with reasonable exceptions, should be 
open and published, revisable and form useful social capital where appropri- 
ate to do so (Trenham and Steer 2019). AI ethics frameworks on data dovetail 
with the requirements of usability, though they do fall short on the nuances of 
respecting interpersonal relationships and community values. There is substan- 
tial overlap between usability and community; and usability and dependence. 
Which is to say, that data must be usable and dependable for the community 
who must access and control it. If ICT systems leave communities dependent 
on the expertise of ‘outsiders’ to maintain them, then they create vulnerabilities 
for their sovereignty. Usability for communities is vital for all kinds of commu- 
nities, from families, hospitals, schools, cultural groups, businesses and organi- 
sations as well as for the nation(-states). 


Politics 


Good Data reveals and challenges the existing political and economic order so 
that data empowered citizens can secure a good polity. Citizen-led data initia- 
tives lead to empowered citizens (see e.g. Valencia and Restrepo 2019). Open 
data enables citizen activism and empowerment (see Gray and Lammerhirt 
2019). Strong information security, online anonymity and encryption tools are 
integral to a good polity. Social activism must proceed with ‘good enough data’ 
(Gutierrez 2019; Gabrys, Pritchard and Barratt 2016) to promote the use of 
data by citizens to impose political pressure for social ends. How can AI con- 
tribute to the empowerment of citizens, without data, models and algorithms 
putting them at risk? AI systems need to be understood by citizens so that 
outputs and recommendations are trusted as working in their favour. To this 
end, the politics pillar on activism for Good Data and good AI is drawn from 
the other three: community, rights and usability. AI infrastructure controlled 
and accessed by communities that progresses their rights and interests is the 
gold standard of genuinely ethical AI. 

Our research into Good Data encourages data optimism beyond minimal 
ethical checklists and duties - thus our aim is supererogatory (Heyd 1982). 
We recommend these Good Data pillars to progress political and social jus- 
tice agendas such as citizen-led data initiatives, accepting ‘good enough’ data to 
achieve aims (Gutierrez 2019; Gabrys, Pritchard and Barratt 2016). The aim is 
to dismantle existing power structures through the empowerment of commu- 
nities and citizens via data and digital technologies and enhancing technologi- 
cal sovereignty (Mann et al. 2020). 


Al Ethics Needs Good Data III 


Moving away from the body of critique of pervasive ‘bad data’ practices by 
both governments and private actors in the globalised digital economy, we 
paint an alternative, more optimistic but still pragmatic picture of the datafied 
future. In order to secure a just and equitable digital economy and society we 
need to consider community, rights, usability and politics. 


AI for Good Data? Good Data for AI? 


But how can we implement these pillars in practice? Can AI be fed with or 
nourished by Good Data? Can AI feed and nourish Good Data? We acknowl- 
edge that here too we are not the first people to consider this issue. For instance, 
Floridi et al. (2020) discuss the emerging ‘AI for Social Good’ (AI4SG) trend 
and formulate seven ‘essential’ but not ‘sufficient’ sociotechnical principles of 
their own for AI4SG. These principles are rather technocratic although Floridi 
et al. (2020) do acknowledge the wider contexts in which AI development and 
deployment take place and the power imbalances which persist forming the 
backdrop to these developments and deployments. We instead seek to centre 
these contexts and power imbalances in proposing Good Data as a frame or 
concept for AI development. 

For AI to be Good Data and Good Data to be AI, AI would need to be built 
for communities using data available and relevant to them. To achieve good 
data, communities need to gather, store and process their own data; they need 
to have access to open and closed data sets of relevance to their interests. Com- 
munities need cloud storage, AI classifiers, data scientists and so forth to build 
the tools communities need to become empowered. It is unclear what socioeco- 
nomic structures would enable genuinely ethical AI with Good Data - but cer- 
tainly not current ones. At the very least it would require massive investment, 
democratisation and reimagining of ICT infrastructure. The most powerful 
produce and selectively hide data. The least powerful depend on data gathered, 
curated and displayed by the empowered, often data about them. 

What, then, are the barriers to achieving Good Data? We have mentioned a 
few in passing above: the smokescreen of ethics to obscure enforceable state- 
led regulation; the limits of law; the broader political economy of neoliberal 
capitalism and corporate greed and its impacts of extractive logic on the nat- 
ural world; and the corresponding power imbalances and inequalities in a 
world characterised by privilege and division. In addition, creating effective 
and trusted AI is elusive and expensive and requires access to valuable data 
sets, knowledge workers as well as access to high quality digital architecture, 
test and evaluation processes and user testing in the anticipated context of use. 
Data and software must be incorporated into secure and compliant back-end 
databases with user-friendly front-end interfaces. While AI is becoming much 
more ‘plug and play, enabling those with less skills to add data to software 
products and curate algorithms, the end-to-end construction of AI products to 
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meet a societal need is still a bespoke and expensive business. The complexity 
of AI in addition to its cost is why AI production is dominated by three kinds 
of enterprise: (a) technical startups, (b) medium to large sized corporations 
and (c) governments. There is comparatively very little AI constructed by low- 
resource community groups, non-profits, aid agencies, advocacy groups for the 
marginalised and disenfranchised. 

What, then, can be done to address these barriers? A multipronged approach 
is necessary to (start to) dismantle (some of) them, with a recognition of the 
limits of law, code, markets and social norms (as per Lessig’s (1999) modes 
of regulation) as tools through which Good Data can be achieved. Central 
though to the problem is the political economy of data in the context of neo- 
liberal capitalism. Both governments and corporations have a strong, in some 
cases existential, interest in gathering, analysing and using data. Truly curbing 
these practices through law or ethics or markets may be extremely difficult to 
achieve in practice absent major societal change. 

However, not to give into defeatism regarding these large challenges, we 
view the way forward as beginning to create an alternative vision of a datafied 
society and economy which promotes and achieves social and environmental 
justice goals, and we view incremental change for now to be the most likely 
pragmatic path in this direction. 

There are some cases of AI for social good, for example, a software engineer 
developed an AI that could automatically write letters for people who received 
parking tickets in a way that got them a waiver from having to pay the fine (Dale 
2016; DoNotPay 2020). The business aims to connect people to legal advice from 
parking tickets to divorce (Krause 2017). The people most likely to benefit 
from such an Al included those in the community who lack the funds to pay 
the parking ticket and may lack the education, literacy, knowledge or experi- 
ence required to negotiate written legal documents. AI products for legal aid 
use AI to improve equity and fairness. There are cases of AI for environmental 
good, such as poaching deterrence and identification of rare and endangered 
species. AI can automatically count flocks, track animals, assess perimeter, 
monitor habitats.' Although the use of AI for surveillance - even for ostensi- 
bly ‘good’ reasons - remains controversial. Consequentialist justifications will 
never satisfy rights-based or duty-based obligations to other humans such as 
protection from persistent surveillance. 

However, AI for social good is ad hoc. That is to say, private individuals gen- 
erate the concept of AI to alleviate some source of injustice and proceed to 
develop a technology that may be useful for a specific purpose, but does not 
have the backing of a significant entity to ensure that AI products are chosen to 
improve quality of life for the most vulnerable through a process of consulta- 
tion and oversight. 

Non-profits and other organisations dedicated to the alleviation of human 
suffering and improving justice are traditionally staffed by less technical per- 
sons, such as those with legal training rather than technical training in software 
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engineering and machine learning. It is difficult to build up the technical com- 
petence required to create AI for social justice within organisations already 
struggling to deliver their organisation’s missions within tight budgets. 

Governments might be good candidates to make AI for the good of all citi- 
zens. However, time and time again governments are found to use citizen data 
for uses that do not align with the values and expectations of marginalised 
groups within society, such as First Nations peoples (e.g., see Kukutai and Taylor 
2016; Lovett et al. 2019), the unemployed or marginally employed (e.g., for 
an overview of Australia’s RoboDebt welfare surveillance program see Mann 
2019; 2020). 

The quality of AI outputs is based on the data that it is fed and curated with. 
Organisations lacking access to large data sets will be unable to participate in 
the AI economy. Conversely, large corporations that focus on data collection as 
a primary asset collect vast data sets to feed AI algorithms. 


Moving Towards ‘Better’ Data in AI 


We view the way forward as beginning to create an alternative vision of a data- 
fied society and economy which promotes and achieves social and environ- 
mental justice goals, and we view incremental change for now to be the most 
likely pragmatic path in this direction. 

As a way forward to ensuring Good Data we look to integrate Lessig’s (1999) 
various approaches or modes of regulating technology, namely: law, code/ 
architecture, social norms and markets with philosophical models of informa- 
tion, acknowledging epistemic, ethical and political conceptions of what con- 
stitutes ‘the good’ In order to ensure technology, such as AI, rests on Good 
Data pillars and exhibits Good Data values, a multipronged approach involving 
these different modalities of regulation and conceptual apparatus is necessary. 

It is insufficient to rely only on formal law to achieve ethical and politically 
progressive outcomes - as also recognised by Kalulé and Joque (2019), Kalule 
(2019), Hoffman (2019) and Amoore (2020). We do not necessarily accept 
the determinist view that the law follows technological innovation which the 
‘regulatory disconnect’ suggests (Brownsword and Goodwin 2012). Indeed, 
we acknowledge that digital technologies have benefitted from emerging in a 
period when deregulatory, neoliberal ideologies have prevailed which has led 
to the law playing ‘catch-up’ (Daly 2016). 

However, we do see some potentially promising provisions such as Article 
25 of the EU’s General Data Protection Regulation on ‘data protection by 
design and default; as an attempt to ‘join up’ law and code in implementing 
ethical principles. But, we and others have questions about how this translates - 
if indeed it is possible to do so - into the design or hardcoding of systems 
(Koops and Leenes 2014), and what the consequences, including unintended of 
such an intention to embed principles into technological systems may be 
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(Ihde 2006). As we also see in the US, the mere fact of AI ethics principles hav- 
ing the binding force of law is insufficient to establish their ‘goodness. 

It is also insufficient to focus solely on social norms in the form of unen- 
forceable ethical principles as these can indeed result in ethics washing. We 
agree with Powles and Nissenbaum (2018) who see a focus on ‘solving’ issues 
of fairness, accountability and transparency as foremost among these discus- 
sions through code (and to some extent social norms) as problematic, and one 
which obscures the broader structural problems which are at the root of bias 
in machine learning. Again, Ihde’s critique of the ‘designer fallacy’ mentioned 
above also applies here regarding the unintended consequences of attempting 
to ‘design out’ bias and other problems in digital technologies. Furthermore, 
broader existential questions about whether a particular system or technol- 
ogy should even be used in the first place are of key importance but also often 
obscured in the focus on issues such as bias (Powles and Nissenbaum 2018). 
A Good Data approach to AI would certainly ask these questions before any 
such system was implemented and see that these problems are pertaining to 
broader social, political and economic contexts which will not easily be ‘solved’ 
by technology alone. 

While it may not eventuate in a Good Data utopia, we view that laws, social 
norms, code and markets ought to promote and attempt to ensure Good Data 
practices. While at least one underlying problem of Good Data may be the capi- 
talist political economy (Daly 2016; Benthall 2018), some incremental steps to 
promote positive change can still be taken (Raicu 2018) - or ‘better’ data. 

Better data for AI can be promoted through a number of ways. While each 
alone is insufficient, together they may equate to progress. The multiplicity of 
ways and methods to achieve better data for AI reflect the embeddedness of AI 
in pre-existing, currently existing and future socio-environmental-economic 
conditions, and as not something that can easily be ‘solved’ via a statement of 
ethics principles. 

Pragmatically, to achieve better data even the market can assist through cor- 
porate social responsibility initiatives by private sector players through ensuring 
they act in ethical ways (beyond legal obligations) in their product develop- 
ment, manufacturing, implementation and sales (Grigore, Molesworth and 
Watkins 2017). Better data for AI can also be advanced by environmentally 
sustainable corporate social responsibility initiatives (see e.g. Chuang and 
Huang 2018) and circular economy initiatives, and by corporations ensuring 
adherence to high labour standards at all stages in the supply chain. The Fair- 
phone is an example of an attempt to produce such an ethical piece of digi- 
tal technology in the private sector (Akema, Whiteman and Kennedy 2016). 
Workers in technology companies can also do what they can to resist Bad 
Data practices, as we have witnessed at Google and Amazon in recent years 
(Montiero 2017; Salinas and D’Onfro 2018). Data ethical norms and practices 
need to be inculcated at all levels of society including formal and informal 
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educational settings; internet and social networking standards, media and 
communication channels and in the attainment of professional accreditation 
and qualifications. While law alone is insufficient, it also should not be dis- 
pensed with as a tool for moving towards better data for AI. 

Moreover, the debate on AI ethics has been dominated by western approaches 
to this topic. We also look to the Indigenous Data Sovereignty movements 
developed and led by First Nations peoples as presenting radically different 
visions of data collection and usage from the hegemonic western norm, and 
bring to the fore key questions of whether data should be collected and by 
whom (Kukutai and Taylor 2016; Lovett et al. 2019). Good Data approaches 
must take account of Indigenous perspectives and worldviews on data and the 
discrimination and oppression that Indigenous peoples and nations have hith- 
erto experienced through western colonialism and imperialism. We are already 
seeing promising developments. New Zealand has recently released a draft 
algorithmic charter that explicitly seeks to ‘embed a Te Ao Maori perspective 
in algorithm development or procurement.’ 


Conclusion 


We have argued here that AI needs Good Data. The four pillars of Good Data: 
community, rights, usability and politics are at the forefront of a just digital 
society and economy. Good Data situates genuinely ethical AI within commu- 
nities and collectives, rather than individuals or large organisations. The well- 
being of the people and environment must be at the forefront of AI ethical 
considerations, including considerations not to use AI at all. 

We have also argued that AI needs Good Data because the issues that are at 
the forefront of the digital society and economy go beyond pre-existing dis- 
cussions of ethics. Like data itself, it is impossible for us to cover everything 
encompassed by ‘Good Data’ and accordingly we cannot offer a ‘complete, 
‘comprehensive’ or ‘perfect’ account of Good Data at this stage (if indeed ever). 
But we can say that Good Data is a more expansive concept which aims to 
encompass practices beyond ‘ethics’ and also human rights, environmental 
and social justice concerns arising around data which may involve extending 
beyond the focus to date on ‘AI ethics’ and an emerging focus on ‘AI law’ to 
address deficiencies with ‘AI ethics’ 

Good Data should permeate digital technology development, implemen- 
tation and use at all stages in the process, and involve different tools, nota- 
bly law, norms, code and markets, in order to bring about ‘better’ - or ‘good 
enough scenarios, even if the broader societal conditions and limitations 
mean that it is difficult to bring about ‘Best Data. Good Data can also involve 
the forbearance from generating and using data, either at all, or in some cir- 
cumstances or by some specific people. This has implications for businesses as 
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data collection, analysis and use should be orchestrated and mediated by, with 
and for data subjects, rather than determined by those in power (corporate 
or otherwise). 

We also hope that Good Data can encompass a more global approach, 
rather than just (re)centring perspectives from the Global North, as already 
noted - and critiqued - by Arora (2016) and Kalulé and Joque (2019). How- 
ever, we acknowledge we are also coming from a northern/western perspec- 
tive ourselves.’ Already there is emerging discussion from China, in particular, 
on technology ethics, and legislative activity in many jurisdictions around the 
world regarding data localisation (Melashchenko 2019). The Indigenous Data 
Sovereignty movements also display different worldviews and approaches to 
issues of data situated in Indigenous laws, cultures and traditions, countervail- 
ing the practices and uses of data by colonial and imperial forces against Indig- 
enous peoples, and representing more perspectives beyond the western focus 
on normativity and ethics as regards technologies including AI. 

Pragmatically, we view the next steps for all involved in the digital society 
and economy (which, in fact, is all of us) as trying to engage and empower each 
other to build Good Data initiatives and communities of change, rather than let- 
ting governments and corporations build a Bad Data future for us. Yet it is also 
important that governments and corporations contribute positively to the Good 
Data future by taking note and implementing ‘good’ and more ethical data prac- 
tices. Only with such a multifaceted approach encompassing will we be able to 
achieve some semblance of Good Data for AI and for the digital more generally. 


Notes 


! From https://www.dronezon.com/drones-for-good/wildlife-conservation 
-protection-using-anti-poaching-drones-technology 

? The Draft NZ Algorithmic Charter is available here: https://data.govt.nz 
/use-data/data-ethics/government-algorithm-transparency-and-accounta 
bility/algorithm-charter 

> Or perhaps more accurately for two of us, a ‘Global North-in-South 
perspective - see Mann and Daly (2019). 
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CHAPTER 8 


Ihe Social Reconfiguration of Artificial 
Intelligence: Utility and Feasibility 


James Steinhoff 


Introduction 


This chapter addresses the notion of ‘AI for everyone’ via the concept of social 
reconfiguration. This was originally formulated by Jasper Bernes (2013) in his 
critique of what he calls the ‘reconfiguration thesis’ or the assumption, held by 
many Marxists and other critics of capital, that ‘all existing means of production 
must have some use beyond capital, and that all technological innovation must 
have ... a progressive dimension which is recuperable. In other words, exist- 
ing technologies which have been produced by capital for the advancement 
of capitalist industry can and should be appropriated and redirected towards 
non-capitalist, democratically-determined and socially-beneficial ends - the 
means of production can and should be seized. 

The reconfiguration of AI is a timely topic because, since 2015, almost all 
the big USA tech companies, such as Google and Microsoft, have announced 
commitments to the so-called ‘democratisation’ of AI. Critiques of such pro- 
grams have already been provided (Garvey 2018; Sudmann 2020; Dyer- 
Witheford, Kjosen and Steinhoff 2019, 52-56) and Marxists have long pointed 
out that capitalism is defined by technology not being democratically con- 
trolled, but rather designed and deployed to serve the interests of one small 
sub-group of the world population, the owners of capital (Braverman 1998). 
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In this chapter, I schematise the concept of reconfiguration with two dimen- 
sions: utility and feasibility. I argue that existing considerations of the recon- 
figuration of AI have primarily focused on utility and have largely neglected 
questions of feasibility. Feasibility is considered primarily in relation to the 
materiality of AI, or its concrete aspects which ‘set constraints on and offer 
affordances for use’ (Leonardi and Barley 2008, 171). By attending to the 
materiality of AI we can see how it differs from traditional, industrial means 
of production. 

The chapter first discusses the contemporary form of AI called machine 
learning and its increasing importance to the tech industry. Then I discuss sev- 
eral aspects of its materiality. Next, I discuss Marxist theories of technology 
and existing evaluations of reconfiguring AI, which focus primarily on utility. 
Then I turn to the question of feasibility. I conclude that the social reconfig- 
uration of AI faces substantial difficulties posed by the lack of visibility and 
non-modularity of AI, but that some promise is to be derived from the data 
commons movement. I suggest that further research on socially reconfiguring 
technology should focus more on feasibility, rather than utility and can begin 
by looking at concrete ways to resist the impositions of AI capital. 


Machine Learning Materiality 
Industry 


Early approaches to AI attempted to automate high-level logical reasoning rep- 
resented in formal languages. Such approaches to AI are called ‘symbolic’ or 
‘good old-fashioned’ AI (Haugeland 1989) and have largely been overshadowed 
by a different approach to AI known as machine learning. Machine learning is 
often anthropomorphised, but it is at base the use of statistical methods, called 
learning algorithms, to find patterns in large datasets. On the basis of these pat- 
terns an algorithm called a ‘model’ is produced which may be used to analyse 
new data (Alpaydin 2014, 2-3). A model thus represents ‘knowledge’ of the pat- 
terns found by the learning algorithm and can be used to make useful analyses 
or predictions. Much of the hype around machine learning derives from this 
automated production of models from data, which Domingos (2015) calls the 
‘inverse of programming’ (6-7). 

Machine learning is being applied almost anywhere electronic data is accessi- 
ble. Brynjolfsson and McAfee (2017) argue that machine learning is a general- 
purpose technology comparable to the combustion engine. While this remains 
to be seen, AI has found diverse applications from recommendation engines 
and targeted advertising to predictive policing software, predictive mainte- 
nance, customer resource management and fraud detection. Capital became 
visibly interested in machine learning around 2015. All the biggest tech com- 
panies in the world have since shifted to Al-intensive directions, including 
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Google, Amazon, Microsoft, Facebook, IBM, Baidu, Alibaba and Tencent. 
Older industrial capitals like Siemens and General Electric have followed suit. 
In addition to these huge companies are a variety of middle sized companies 
and an array of startups. Investment in AI startups increased from $1.3 billion 
in 2010 to over $40.4 billion in 2018 (Perrault et al. 2019, 6). This money trick- 
les down to some, but not all, workers involved in producing Al. Salaries for 
machine learning scientists and engineers average $100,000 to $150,000 USD, 
with lavish benefits (Stanford 2019) while essential data-preparing ‘ghost work- 
ers are precariously employed through platforms like Amazon Mechanical 
Turk and are minimally remunerated (Gray and Suri 2019; Li 2017). 


Data 


Perhaps the most fundamental aspect of the materiality of machine learning 
is that it requires a lot of data from which to extract patterns (Alpaydin 2014, 
1-4). One can get an idea of the requisite quantities by looking at some popu- 
lar datasets. The dataset MNIST, a collection of handwritten digits, contains 
70,000 images. Compare it to ImageNet, comprising 14,197,122 images labelled 
with categories and subcategories. The category ‘person’ has 952,000 images 
and 2,035 subcategories (ImageNet 2010). ImageNet is dwarfed by the Gmail 
Smart Reply training set which contains 238,000,000 examples, and the Google 
Books Ngram set which amounts to 468,000,000,000 examples. Google Trans- 
late is said to employ a dataset numbering somewhere in the trillions (Google 
2019). Machine learning is no more than the sophisticated recognition of pat- 
terns across such large datasets (for a sober walkthrough of this process see 
Broussard (2018, 87-120)). 

The companies that produce machine learning commodities are unsurpris- 
ingly concerned with obtaining vast quantities of diverse data. It is no coinci- 
dence that the major producers of AI operate a variety of platform business 
models in which, by acting as intermediaries between users, they can appropri- 
ate all kinds of data (Srnicek 2017). However, quality, as well as quantity, of data 
is important. Data does not come ready-to-use and requires labour intensive 
formatting, cleaning and labelling (Gitelman 2013; Gray and Suri 2019). 


Compute 


Producing machine learning systems requires powerful computing hardware. 
Since few companies can afford to buy such hardware, most advanced machine 
learning models are trained and deployed through the cloud platforms of the 
tech giants. Amazon Web Services dominates the market, but Google, Micro- 
soft, IBM and Baidu all have their own cloud platforms. Computational power 
required for both training and deploying machine learning models is con- 
tinually increasing. The amount of computing power used in the largest AI 
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training runs increased 300,000 times from 2012 and 2018, with no end in 
sight (Amodei and Hernandez 2018). 

Such computation is energy intensive. Cloud platforms thus rely on access 
to energy infrastructures. According to Pearce (2018), the largest data cen- 
tres consume as much power as a city of a million people, and in total, data 
centres consume ‘more than 2 percent of the world’s electricity and emit roughly 
as much CO; as the airline industry. Google, Microsoft and Baidu derive 15%, 
31% and 67% of their energy, respectively, from coal (Cook 2017, 8). Efforts to 
‘green the cloud by increasing renewable energy sources are ongoing, but many 
such campaigns consist of offsetting or buying carbon credits and do not actu- 
ally mean that clouds are contributing less to CO2 production. 


Distribution 


Machine learning requires sources of data, such as social media platforms. Nei- 
ther can it function without storage for data, the substantial work which goes 
into preparing data, nor the cloud or energy infrastructures. Contemporary AI 
is thus not a discrete technological artifact. Even a relatively simple AI product, 
such as a smart home speaker, draws on a ‘vast planetary network’ of labour, 
data, resources and technologies (Crawford and Joler 2018). Machine learning 
cannot be analytically separated from the globally distributed infrastructure, 
both technical and human, on which it relies. And it is perhaps on its way to 
itself becoming another layer of infrastructure. Science and technology studies 
scholars have demonstrated how as infrastructures mature, they become ‘ubiq- 
uitous, accessible, reliable, and transparent’ (Edwards et al. 2007, i). Although 
machine learning as yet possesses none of these qualities perfectly, it is inte- 
grated into many people's daily lives in ways that increasingly approach them. 
Advocates of the AI industry are already positioning AI as a utility compara- 
ble to electricity or the internet, a kind of ‘cognition on tap’ (Steinhoff 2019). 
This effort to cast AI as something immediately available everywhere for users 
is complemented by a technological effort, dubbed ‘democratisation, to make 
the production of AI available to a wider range of users. 


Democratisation’ 


The tech giants have, since around 2015, extolled the ‘democratization’ of 
AI. According to Microsoft CTO Kevin Scott, this means ‘making sure that 
everyone has access to those platforms so that they can use the techniques of 
AI to enhance their own creativity, to build their own businesses, to do their 
jobs’ (Agarwal 2019). For Madhusudan Shekar, Head of Digital Innovation 
at Amazon Internet Services, the democratisation of AI ‘is about making the 
tooling and capabilities of AI/ML available to developers and data scientists at 
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various levels of competence so that anybody can use Alto increase the velocity 
of new customer acquisition, reduce costs and look for new business innova- 
tions’ (quoted in Ananthraj 2019). In general, democratisation efforts take the 
form of more or less automated, cloud-based tools and libraries (some of which 
are free or open source) which either assist developers with building machine 
learning models or help unskilled users incorporate premade models into other 
media. A much-feted early episode of democratisation occurred in 2015 when 
Google open-sourced its now widely-used TensorFlow library. 

Garvey (2018) correctly points out that claims of a ‘democratization’ of 
AI draw on ‘explicitly political language but do not specify whether or how 
such programs are ‘informed by political conceptions of democratic govern- 
ance’ (8079). The idea appears to be that simply distributing the tools consti- 
tutes the democratisation of AI. But this neglects consideration of how the 
AI products which people encounter in their daily lives are not produced or 
deployed through processes of democratic deliberation. Nor do such formula- 
tions address the larger issue of how the capitalist mode of production itself is 
premised on the exclusion of certain stakeholders from social decision-making 
processes. Marxists have discussed this via the distinction between the capital- 
ist (who own and control the means of production) and working classes (who 
own only their ability to labour). A real democratisation of AI would require 
that not only capital controls its development and deployment. 


Capital, Labour and Machines 


Marx held that the relation between the capitalist and working classes was nec- 
essarily antagonistic. Capital has one primary goal: to increase. Marx (1990) 
called this valorisation (293). Mechanisms for valorisation vary across the his- 
torical permutations of capitalism, but all rely on the exploitation of labour 
and the capture of surplus-value. While capitalists and functionaries of capital 
may argue that the valorisation of capital is co-extensive with social flourish- 
ing, the COVID-19 pandemic of 2020 laid the antagonism bare, with CEOs like 
Jeff Bezos and political functionaries like former US President Trump openly 
willing to sacrifice workers for the generation of surplus-value. It appears 
increasingly obvious that, as Land (2017) has argued, capital has ‘no conceiv- 
able meaning beside self-amplification. 

While the interests of labour in the context of work typically take the form 
of better wages and working conditions, the broader interests of labour are the 
interests of socially-existing humans as such and are therefore not amenable to 
a priori description. Marx (1993) describes the ultimate interests of labour as 
the ‘absolute working-out’ of ‘creative potentialities’ (488). Class antagonism 
results from labour’s broad horizon of self-development encountering the nar- 
row logic of capital. Labour might flourish in any number of ways not condu- 
cive to valorisation, so capital ‘systematically selects for human ends compatible 
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with its end ... and systematically represses all human ends that are not’ (Smith 
2009, 123). One of its most effective means for doing so are machines. Driven 
by competition and class struggle, capital introduces machines to increase pro- 
ductivity by cheapening labour power, increasing control over, and dispensing 
with, labourers. 

But this is not to say that Marx considered technology inherently opposed 
to labour. On the contrary, Marx held that human flourishing could only be 
achieved with the help of machines. For Marx and Engels (1969), communism 
only becomes possible when ‘machinery and other inventions [make] it possible 
to hold out the prospect of an all-sided development, a happy existence, for all 
members of society. However, before communism can be attempted, machin- 
ery must be wrested from capital. Workers must seize the means of production, 
or ‘overthrow ... the capitalists and the bureaucrats [and] proceed immediately 
... to replace them in the control over production and distribution, in the work 
of keeping account of labor and products’ (Lenin 1918). Machines thus are nei- 
ther inherently wedded to capital nor labour, but are rather a medium for their 
antagonistic relation. 

Since the valorisation of capital can and often does run orthogonal to the 
interests and wellbeing of labour and since most AI research and production 
today is conducted by capital, one can assume that it is largely conducted in 
accord with the exigencies of valorisation. In other words, AI predominantly 
takes a commodity form (i.e., is designed as something which can be sold 
for profit) or a form which can otherwise augment the valorisation of capital 
(i.e., harvesting user data for inputs). There is no reason to assume that AI 
as a means for capital valorisation stands to benefit society beyond capital. 
Therefore, consideration of ‘AI for everyone’ needs to consider how control 
over AI might be taken away from capital and transferred to a democratic pub- 
lic. If AI is to be directed towards democratically determined ends, it will first 
have to be seized, in the sense that Marxists have talked of seizing the means 
of production. 


Reconfiguration and Artificial Intelligence 


Bernes (2013) defines the ‘reconfiguration thesis’ as the assumption that ‘all 
existing means of production must have some use beyond capital, and that 
all technological innovation must have ... a progressive dimension which is 
recuperable. Bernes first raised the notion of reconfiguration in an analysis of 
capital’s logistics networks. In the course of his argument, Bernes interweaves 
the increasingly logistical nature of capitalism, critical theory, and how it can 
arise from workers who inhabit logistical sites of struggle. In stark contrast 
to his wide-ranging discourse, I will focus narrowly on the notion of recon- 
figuration. We can schematise reconfiguration with two dimensions: utility 
and feasibility. 
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Utility 


A first step to thinking about the potential utility of a reconfigured technology 
is to consider how it is useful now, and to whom. Bernes (2013) argues that 
logistics is ‘capital’s own project of cognitive mapping’ because it allows capital 
to keep track of its dispersed moving parts. It enables a new emphasis on circu- 
lation characterised by practices such as outsourcing, just-in-time production 
and the global arbitrage of commodities, including labour-power. It allows the 
segmentation and stratification of labour, and the brutal creation of ‘sacrifice 
zones’ free of labour regulations (Hedges and Sacco 2014). The utility of logis- 
tics for capital is thus ‘exploitation in its rawest form’ (Bernes 2013). This is not 
likely a use-value for a socially reconfigured AI. 

Andrejevic (2020) argues that under capital, what he calls ‘automated media 
(including AT) tendtowards the automation of subjectivity itself(129). Andrejevic 
argues that this is ultimately impossible on psychoanalytic grounds, but the 
argument that the ultimate end of capitalist AI is the emulation of subjectivity 
has been advanced by others. Land (2014) holds that capital and artificial intel- 
ligence possess a ‘teleological identity’ and that a perfected capitalism will dis- 
pense with human labour for a full-machine economy. Such speculations range 
afield from this paper, but they reinforce the more immediate utility of AI for 
capital. AI is an automation technology with diverse applications for reducing 
and/or eliminating labour costs and implementing new forms of control over 
labour processes and social relations. It was these use-values for capital that the 
earliest Marxist analyses of AI reacted to. In the 1980s, AI was first commer- 
cialised in the form of ‘expert systems’ intended to capture and automate the 
knowledge and reasoning of skilled workers (Feigenbaum, McCorduck and Nii 
1989). Most Marxists of this era were not interested in reconfiguring AI. The 
near consensus was that AI heralded a new wave of deskilling and concomitant 
automation, aimed at cognitive, as well as manual, forms of labour (Cooley 
1981; Athanasiou 1985; Ramtin 1991). 


Planning 


However, another strand of Marxist thought saw utility in reconfigured tech- 
nologies of automation like AI and cybernetics. Both the USSR (Peters 2016) 
and socialist Chile (Medina 2011) attempted to apply cybernetics to solve the 
‘socialist calculation problem, as the economist Ludwig von Mises described 
it. Von Mises (1935) contended that the distribution of resources in a planned 
economy requires an infeasible amount of calculation and that a capitalist mar- 
ket economy achieves this automatically through the market and price system. 
While the attempts at planned economies by Chile and the USSR failed due to 
the primitive computers available at the time, some Marxists have continued 
to pursue the idea of automated economic planning. 
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Cockshott (1988) argued that heuristic processing techniques ‘developed in 
artificial intelligence can be applied to solve planning problems with economi- 
cally acceptable computational costs’ (1). More recently he has described big 
data and supercomputers as the ‘foundations of Cyber Communism (Cockshott 
2017). Others have pointed out that algorithmic technologies for processing 
vast quantities of economic data have already been developed by large corpora- 
tions like Walmart and Amazon (Jameson 2009; Phillips and Rozworski 2019). 
Beyond the processing of economic data, Dyer-Witheford (2013) has suggested 
that AI could be used to lessen bureaucratic burdens: democratic processes 
might be ‘partially delegated to a series of communist software agents ... run- 
ning at the pace of high-speed trading algorithms, scuttling through data rich 
networks, making recommendations to human participants ... communicating 
and cooperating with each other at a variety of levels’ (13). 

Bernes (2013) argues that such positions assume that ‘high-volume and 
hyper-global distribution’ possess ‘usefulness ... beyond production for profit. 
For instance, a society not structured around commodity production would 
not be driven to implement planned obsolescence, so one can imagine that 
the overall volume of things that need to be shipped across the world would 
decrease substantially. In addition, more localised systems of production might 
obviate much of the need for vast planning techniques. The broader point is 
that the utility of a given existing technology for socially-determined, non- 
capitalist ends is not a given if it was built by capitalist firms to advance val- 
orisation. Utility therefore ‘needs to be argued for, not assumed as a matter of 
course (Bernes 2013). 


Full Automation 


Some Marxists have also speculated on the use of AI to eliminate work. This 
line of thought derives from Marx’s notion that ‘the true realm of freedom 
has its ‘basic prerequisite’ the ‘reduction of the working-day’ (Marx 1991, 959). 
Thinkers in the USSR held that automation had a ‘crucial role in the creation 
of the material and technical basis of communist society’ (Cooper 1977,152). 
Since the mid-2010s, a group of Marx-influenced thinkers referred to variously 
as left accelerationism (Srnicek and Williams 2015), postcapitalism theory 
(Mason 2016) and fully automated luxury communism (Bastani 2019) have 
renewed support for such ideals. I refer only to the left accelerationists here, but 
all of these thinkers are united in calling for full automation. 

Left accelerationists argue that under capital, ‘the productive forces of technol- 
ogy’ are constrained and directed ‘towards needlessly narrow ends’ (Williams 
and Srnicek 2014, 355). The technology developed by capital should be seized: 
‘existing infrastructure is not a capitalist stage to be smashed, but a springboard 
to launch towards post-capitalism’ (Williams and Srnicek 2014, 355). They 
hold that ‘existing technology [can be] repurposed immediately’ (Srnicek and 
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Williams 2015). Alongside decarbonising the economy, developing renewable 
energy sources, cheap medicine and space travel, they advocate ‘building artifi- 
cial intelligence’ (Srnicek and Williams 2015). For left accelerationists, a recon- 
figured Al is useful primarily in that it could contribute to full automation, which 
is desirable because ‘machines can increasingly produce all necessary goods 
and services, while also releasing humanity from the effort of producing them’ 
(Srnicek and Williams 2015). Eventually, a ‘fully automated economy’ could 
‘liberate humanity from the drudgery of work while simultaneously producing 
increasing amounts of wealth’ (Srnicek and Williams 2015, 109). 

The automation of bad work and the administration of a planned economy 
are certainly useful applications of AI that extend beyond the logic of valorisa- 
tion. But utility should be considered alongside feasibility. 


Feasibility 


Even if a given capitalist technology presents useful possibilities, it is not neces- 
sarily the case that its social reconfiguration appears feasible. Bernes presents 
several reasons why a social reconfiguration of logistics is infeasible, two of 
which derive from its distributed nature, and are likewise applicable to contem- 
porary machine learning. The first of these pertains to visibility. 


Visibility 


Logistics comprises a vast, heterogeneous network of technologies and insti- 
tutions which remains invisible as a whole to the workers who populate its 
variegated zones. The means of logistical production are distributed across this 
network, but ‘[o]ne cannot imagine seizing that which one cannot visualise, 
and inside of which one’s place remains uncertain (Bernes 2013). Logistics is 
capital’s means for knowing itself, but this knowledge is barred from workers. 
This sense of visibility is not only an issue when considering the initial seizure 
of a technology, but also for tracking the progress of its social reconfiguration, 
which is unlikely to occur instantly. To persevere, ‘struggles need to recognise 
themselves in the effects they create, they need to be able to map out those 
effects ... within a political sequence that has both past and future, that opens 
onto a horizon of possibilities’ (Bernes 2013). Contemporary AI presents simi- 
lar problems of visibility, from several different angles. 

AI is temporally and physically distributed across layers of infrastructures. 
To ‘see’ AI we need ways to chart this vast network and make it appear as a 
coherent collection of people and things. Excellent work on visualising AI has 
been done in visual essays by Crawford and Joler (2018), which reveals the 
diverse materiality of AI, and by Pasquinelli and Joler (2020) which aims to 
‘secularize’ AI by casting it not as alien intelligence, but something more like an 
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optical instrument, akin to a microscope. But visual essays can only go so far. In 
a more fundamental sense, visibility is a problem of knowledge. 

‘Democratisation of AI programs aim to make AI accessible to less skilled 
users, but they do so by abstracting from the underlying code with user- 
friendly interfaces. Of course, all computing technology today uses layers of 
abstraction, whether to allow skilled users to achieve complex ends more easily 
or to allow less-skilled users to do something at all (including the word proces- 
sor I am using to write this chapter). Not many people write machine code. But 
as Kittler (1995) pointed out, increasing layers of abstraction from the underly- 
ing materiality of the computer mean that the potential ends it might be put 
to are reduced; layers of software act as a ‘secrecy system’ blocking access to 
basic functionality. So-called democratised’ machine learning does not enable 
the production of novel applications of the technology beyond pre-determined 
bounds. At best, it allows more users to apply pre-canned software tools. 

Further, while the open sourcing of AI tools and libraries like Google’s Ten- 
sorFlow may seem like a truly democratic move insofar as companies are giv- 
ing away proprietary software, it also has competitive dimensions motivated 
by valorisation. Open sourcing can generate a community around the software 
which entails skilled developers (and potential future employees) for the com- 
pany who produces the software. It can also create a software ecosystem based 
on those tools, which a company can retain control over through a variety 
of mechanisms from mandatory lock-in agreements to closed source variants of 
programs. Google used (and uses) such strategies to make Android the most 
popular mobile operating system in the world (Amadeo 2018). While Google’s 
TensorFlow can currently be run on competing clouds, there are indications 
that the tech giants are aiming towards fully siloed AI ecosystems. Google is not 
alone in developing proprietary hardware specially designed for AI. Google's 
Tensor Processing Unit (TPU) provides a ‘performance boost’ over traditional 
hardware, but ‘only if you use the right kind of machine-learning framework 
with it... Googles own TensorFlow’ (Yegulalp 2017). Open source AI software 
is thus one tactic of a larger strategy by which AI capitals combat their rivals for 
a share of surplus-value. 

Visibility is also a technical problem. Machine learning has a ‘black box’ 
problem because the complex computations that occur within a system can- 
not be disassembled and examined and thus its outputs remain inexplicable. 
As one researcher puts it, the: ‘problem is that the knowledge gets baked into 
the network, rather than into us’ (quoted in Castelvecchi 2016). Even if some 
machine learning models could be reconfigured without being rebuilt, their 
operations would remain inscrutable, presenting problems of accountability 
(Garigliano and Mich 2019). The delegation of economic planning or bureau- 
cratic decision-making to a black box might be tolerable for some, as long as no 
mistakes are made, but it seems dubious that such occult mechanisms would 
represent a substantial improvement for democratic decision-making over del- 
egating social decisions to the so-called logic of the market. 
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Non-modularity 


A second dimension of feasibility also concerns distribution, but from a tactile, 
rather than visual, standpoint. Bernes (2013) argues that while revolutions are 
necessarily localised, ‘any attempt to seize the means of [logistical] production 
would require an immediately global seizure. Without connection to the rest of 
the logistical network, a reconfigured port facility is of little use. On the other 
hand, maintaining connection with the rest of the network entails ‘trade with 
capitalist partners, an enchainment to production for profit ... the results of 
which will be nothing less than disastrous’ (Bernes 2013). One might reply that 
taking the whole system over at once is not necessary - one can appropriate it 
piecemeal. This might be the case, but it needs to be taken into account that 
infrastructures are built on top of infrastructures and intertwined with them in 
‘recursive ways (Larkin 2013, 30). A technology that is part of a larger system 
may not necessarily be possible to reconfigure by itself. 

In a second consideration of the problem of reconfiguration, focused this 
time on agriculture and energy, Bernes discusses the non-modularity of cer- 
tain technologies. By this he means technologies that ‘fit together into technical 
ensembles that exhibit a strong degree of path-dependency, meaning historical 
implementation strongly influences future development, precluding or making 
difficult many configurations we may find desirable’ (Bernes 2018, 334). He 
singles out energy infrastructure as particularly non-modular and argues that 
hopes of simply substituting clean energy sources, even if all political oppo- 
sition were removed, is wishful thinking because the ‘technology [we] would 
inherit works with and only with fossil fuels’ (Bernes 2018, 334). 

To consider the non-modularity of machine learning, recall its reliance on the 
highly centralised clouds maintained by the tech giants. Any reconfiguration of 
AI would require a seizure of the data centres which make up the cloud as well 
as the energy sources and infrastructures necessary to power them. Such facili- 
ties could, certainly, be seized like more traditional means of production, such 
as factories. But this presents its own host of material problems. One concerns 
the powerful hardware required for AI and its energy consumption. While 
some greening of data centres is evidently possible, it is uncertain whether 
greening efforts can keep pace with the increasing computational demands of 
machine learning. Developers at OpenAl recently stated that ‘it’s difficult to be 
confident that the recent trend of rapid increase in compute usage will stop, and 
we see many reasons that the trend could continue’ (Sastry et al. 2019). 

Cutting-edge machine learning is increasingly out of reach for organisations 
without resources on par with Facebook or Google. OpenAI was founded as a 
non-profit research lab with substantial donations from the likes of Elon Musk 
and Peter Thiel, but in 2019, justified a switch to a ‘limited profit’ model, in 
partnership with Microsoft, because AI research ‘requires a lot of capital for 
computational power’ (Brockman 2019). If contemporary machine learning 
algorithms are indefinitely scalable, meaning that their performance improves 
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as long as more data and computational power are made available, then the 
hardware cost of AI research and development will continue to rise. If the 
reconfiguration of Al is to occur in a local context, and if it wishes to remain on 
functional par with capitalist AI, it will have to devote considerable resources 
to the requisite hardware and figure out how to make their operation more 
ecologically feasible. 

But perhaps seizing data centres is not necessary for the reconfiguration of 
AI. Some commentators hope to shift the computational load from the central- 
ised cloud onto individual devices in a technique called decentralised or edge 
computing. While increasing amounts of edge computing seem likely as com- 
ponents continue to decrease in size, data centres will always offer more space 
and thus more total computing power. The expert consensus seems to be that 
with existing technology it is ‘not possible to move Cloud-levels of compute 
onto the edge’ (Bailey 2019). Another alternative to seizing the existing cloud 
could be to construct an alternative cloud. Such initiatives exist, such as the 
CommonsCloud Alliance, which aims to build a cloud based not on centralised 
data centres, but on computing power and storage space shared amongst users 
(Sylvester-Bradley 2018). This seems feasible, but unlikely to compare to the 
capacities of the clouds of the tech giants. 

Data itself also raises several questions of feasibility. The first pertains to 
data collection. Many AI systems are trained on publicly available datasets in 
early stages of development, but usually, proprietary datasets are necessary to 
complete a project (Polovets 2015). The preparation and labelling of these is a 
labour-intensive and time-consuming process (Wu 2018). Creating a dataset 
also requires a venue for data collection in the first place. Companies such as 
Amazon and Google harvest reams of data from the interactions of users of 
their applications, even when they claim not to be, as smart home devices have 
shown (Fingas 2019). One business analysis of IBM suggests that because the 
company lacks a data collection venue, it will face difficulties developing its 
AI endeavours (Kisner, Wishnow and Ivannikov 2017, 19-20). This perhaps 
indicates why, in 2020, IBM entered into partnership with data-rich enterprise 
software company Salesforce. AI entails a capitalism built around surveillance, 
enabling ‘data extractivism (Zuboff 2019). How desirable is pervasive, multi- 
modal surveillance for a socially reconfigured AI? 

Machine learning’s reliance on data also necessitates a unique form of main- 
tenance. A model which functioned well when it was deployed will no longer 
do so if the domain it is applied to changes such that the data it was trained on 
no longer accurately reflects that domain (Schmitz 2017). Imagine a hypotheti- 
cal model trained to recognise traffic signs. If overnight the red octagons read- 
ing STOP were replaced with purple triangles reading HALT, the model would 
no longer function and would require maintenance. A social reconfiguration 
of AI will presumably be one component of a larger democratic restructur- 
ing of society with substantial changes to the normal routines of social life. A 
preview of this sort of disruption for AI has been provided by the COVID-19 
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pandemic, ‘models trained on normal human behavior are now finding that 
normal has changed, and some are no longer working as they should’ (Heaven 
2020). When substantial shifts in human behaviour occur, models no longer 
map onto reality. It is reasonable to assume that models trained on data pro- 
duced by life under capital may not function in a society striving to fundamen- 
tally change its basic axioms. 

There is, however, at least one reason for optimism concerning data. A prom- 
ising alternative to mass surveillance and siloed data ecosystems comes from 
the notion of data commons, in which individuals and institutions share data 
willingly, with controls over anonymity and a goal to make data valuable not 
only to tech companies, but also to its producers. The DECODE projects in 
Barcelona and Amsterdam have piloted aspects ofa data commons successfully 
and are planning to scale up in the future (Bass and Old 2020). An interesting 
aspect of these projects is their use of other relatively new technologies, such 
as smart contracts (Alharby and Van Moorsel 2017), to aggregate and analyse 
sensitive data in ways which preserve privacy and retain user control. These 
projects provide a concrete demonstration of the feasibility of reconfiguring 
some aspects of data ecosystems. That they draw on novel smart contracts 
should remind us that assessments of feasibility are necessarily contextual; they 
are constrained by the knowledge of the assessor and the current technological 
milieu. As such, this chapter makes an argument which remains open to revi- 
sion. Any social reconfiguration of AI will have to go beyond the assessment 
attempted here and search out such novelties, technological or other, as might 
be relevant. 


Conclusion: Counter-Al 


Discussing the democratization of AI, Kevin Scott, CTO at Microsoft, makes 
the following comparison with the industrial revolution: 


the people who benefited from [steam powered] technology were folks 
who had the capital to ... build factories and businesses around the 
machines and people who had expertise to design, build and operate 
them. But eventually ... the technology democratized. You don't get 
any sort of advantage now, as a capital owner, because you can build 
an engine. And what we ... need to do ... is dramatically contract that 
period of time where Al is so hard to do that only a handful of people 
can do it. (Agarwal 2019) 


Scott’s sense of democratisation here hinges on the mere generalisation of a 
technology. The notion seems to be that because, over time, knowledge of 
how to build steam engines diffused through the population, this technology 
became democratised - any ‘capital owner can build or go out and buy a 


136 Al for Everyone? 


steam engine. But this formulation seems blissfully unaware of the inequali- 
ties between capital owners and labour and thus it precisely misunderstands 
the meaning of democratisation. There are a lot of people in the world without 
any capital at all. Further, the mere distribution of free AI tools does not ensure 
democratic control over the centralised means of AI production nor upset the 
advantage held by the current producers of AI. This chapter has thus explored 
what it might mean to actually democratise AI, or rather, to socially recon- 
figure existing AI into an ‘AI for everyone. The central point I have hoped to 
make is that consideration of the utility of a socially-reconfigured AI should 
be complemented by consideration of feasibility, which is largely determined 
by the ‘material character of the powers and forces’ involved in the technology 
(Bernes 2018, 336). 

Reconfiguring AI entails simultaneous reconfiguration of large chunks of the 
tech sector, energy infrastructure, advertising industry, data market/ecosystem, 
and also requires social deliberation over aspects of the material character of 
AI, such as its apparent need for surveillance. This assessment resonates with 
that of Huber (2020), who lucidly argues that a reconfiguration of the capital- 
ist food industry is impossible via incremental piecemeal tweaking, but will 
instead require revolutionising the entire system. Morozov (2019) makes the 
same case for the ‘feedback infrastructure’ or the means of producing, harvest- 
ing and processing data which are so essential to AI. On the other hand, the 
data commons movement indicates practical ways in which the data which 
machine learning systems are built from can be utlised to benefit those not at 
the helms of big tech capitals. While the data commons movement is occurring 
within the circuits of capital, it shows how a social reconfiguration of AI might 
begin. Even if the seizure of AI seems herculean, data commons projects dem- 
onstrate a concrete modicum of feasibility. 

Finally, if one cannot seize AI today, one can still resist it. It is true, however, 
that resistance is a wearily overused term for critics of capitalism. What does 
it mean, in practice, to resist capitalist machine learning? For a final time, I 
will draw on Bernes (2013), who suggests that we might imagine a ‘logistics 
against logistics, a counter-logistics which employs the conceptual and techni- 
cal equipment of the industry in order to identify and exploit bottlenecks ... 
This counter-logistics might be a proletarian art of war to match capital's own 
ars belli. While this chapter cannot adequately explore the idea, it can suggest 
that there could be a proletarian counter-AI built around the axis of data on 
which machine learning, and the capital it increasingly powers, runs. 

Early forms of this might take the form of rendering data unavailable or 
unusable to capital. Users might engage in ‘data strikes’ by deleting or other- 
wise denying access to their data (Vincent, Hecht and Sen 2019) and they might 
distort or ‘poison’ their data by introducing inaccurate or harmful patterns into 
it (Vincent et al. 2021). But what about the data infrastructure more broadly? 
It should be possible to determine key bottlenecks in the valorisation processes 
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of capitalist AI, at which democratic control might be one day exercised, but 
for now might provide at least an opening for proto-democratic intervention. 
However, since secrecy is a prime virtue of AI capital, it can be difficult to 
obtain information on its data-intensive processes. One might thus look into 
the technical literature on AI for concerns which might be exploited by those 
resisting capitalist AI, such as ‘adversarial attacks’ which exploit the pattern 
recognition properties of machine learning to render model output inaccu- 
rate (Samangouei, Kabkab and Chellappa 2018). However, technical problems 
need to be considered in relation to how they are implicated within valorisa- 
tion processes. Thus, a fruitful direction for research is business-oriented lit- 
erature on AI adoption and production. This is generated by the producers of 
AI commodities - Microsofts online AI Business School and Google’s array 
of free AI education courses are two examples - but also by a wide variety of 
industry promoters, from consulting firms, see Accenture’s (n.d.) guide to ‘AI 
for Business Transformation and management-oriented books like The Execu- 
tive Guide to Artificial Intelligence (Burgess 2018). Such sources can reveal what 
AI capitals are worried about and indicate potential bottlenecks amenable to 
outsider intervention. Once identified, bottlenecks can be analysed and the 
social relations which support valorisation via AI therein might be replaced 
with alternative social relations not amenable to the data-hungry valorisa- 
tion of AI capital. Finding bottlenecks returns us to the question of visibility, 
without which strategy cannot be formulated. I hope this chapter will contrib- 
ute to an incremental increase in visibility and, perhaps, a half step towards 
a strategy. 
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CHAPTER 9 


Creating the Technological Saviour: 
Discourses on Alin Europe 
and the Legitimation of Super Capitalism 


Benedetta Brevini 


Introduction 


Dominant narratives in public fora, and increasingly within governments, 
place great importance on nations achieving leadership in artificial intelligence 
(AT). What is becoming clear is that world leaders are invested in making AI 
the business opportunity of the future - and thereby selling it as a virtue and a 
public good (Economist 2017, 2018; World Economic Forum 2018). 

Scholars in political economy of communication have shown how discourses 
around digital technologies have historically been constructed around modern 
myths (Mosco 2004) with major references to utopian worlds and possibili- 
ties. Myths, conceived as the dominant ideologies of our time (Barthes 1993) 
become powerful devices that normalise conventional wisdom into common 
sense’ (Gramsci 1971), thus making the conception of alternatives virtually 
impossible. As a result, digital developments and policies are adopted without 
the benefit of an informed debate (Brevini 2020). 

Europe is rarely considered a leader in AI developments, but rather, seems to 
struggle to find its own voice squeezed between China and the United States. 
However, 2018 was a crucial year in Europe for the advancement of national 
and EU strategies on AI. The journey to develop an AI strategy in the EU 
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started in April 2018, when the European Commission presented the ‘Declara- 
tion of Cooperation on AT now signed by all 28 Member States including Nor- 
way. Member states pledged to work together towards ‘a comprehensive and 
integrated European approach on AT (EU Declaration 2018). The declaration 
was followed by two communications reports by High-Level Expert groups 
on AI (High-Level Expert Group 2019); and a White Paper was published in 
February 2020. 

This chapter aims to unwrap the recurrent myths employed in discourses 
on AI in Europe. In doing so, this work aims to embrace a research agenda 
that integrates political economy (Mosco 2004; Fuchs 2015; McChesney 2013) 
with cultural analysis, thus considering the idea of myth and mythmaking as an 
essential dimension of inquiry. 


Why Dominant Discourses Are Crucial 


That technology discourses have a central role in the legitimation of a specific 
political economic order has been at the centre of scholarship debates for some 
time (Mosco 2004; Freedman 2002; Brevini 2020). For example, Fisher (2010) 
has shown how technology discourse legitimated the ‘post-Fordist phase’ of 
capitalism characterised by ‘the weakening of labour and the state vis-a-vis 
capital, the liberalization of markets, the privatization of work, and the flexibi- 
lization of employment’ (Fisher 2010, 234). After all, technological ‘fixes’ have 
historically been crucial to solve potential barriers to capital accumulation. As 
David Harvey (2005) argues, technology becomes ‘a prime mover’ of capital- 
ist growth (Harvey 2005). Likewise, several studies in the fields of history of 
technology, sociology and political economy of communication have shown 
the ideological functions of technology discourse (Mosco 2004; Barbrook and 
Cameron 1996; Dean 2002). 

In this chapter I am drawing, in particular, on analysis that recognises the 
crucial role of myths in building discourses. In the Digital Sublime (2004) Mosco 
explains how myths are used to claim how digital technology is capable of trig- 
gering an historical break: ‘Almost every wave of new technology, including 
information and communication media, has brought with it declarations of the 
end ... Since these tend to take place with no reference to similar proclamations 
in the previous wave, one cannot help but conclude that the rhetoric of tech- 
nology, the technological sublime that David Nye so perceptively identifies, is 
powerful enough to create a widespread historical amnesia (Mosco 2004, 117). 

There are three crucial ways in which myths are used in the context of legiti- 
mising the status quo. Firstly, they are used as a weapon to control political 
debates. Secondly, they are used to depoliticise discourses that would otherwise 
show their contested political character. Thirdly, they are a crucial component 
of hegemonies, thus making it difficult for a counter-hegemonic discourse to 
arise. Barthes (1993) clearly elaborated on this conjoint relation between myths 
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and the political construction of reality in contemporary democracies. Accord- 
ing to Barthes, myths have the ability to construct ‘common sense; thus favour- 
ing established relations of power. But it was Gramsci (1971) who provided 
the crucial link between common sense and discourse. For Gramsci, myths are 
essentially common sense, defined as ‘not something rigid and immobile, but 
... continually transforming itself, enriching itself with scientific ideas and with 
philosophical opinions that have entered ordinary life. Common sense creates 
the folklore of the future, that is a relatively rigid phase of popular knowledge 
at a given place and time’ (Gramsci 1971, 326). Through this process, the val- 
ues of powerful elites are naturalised, becoming the default position against 
which, all things are assessed and compared. Thus, myths, here conceptualised 
as common sense can influence and shape discourse and policy making in way 
that, as Wyatt notes: ‘sometimes today’s imaginary becomes tomorrow's lived 
reality’ (Wyatt 2004, 244). It is through the legitimation of dominant discourses 
(Brevini and Schlosberg 2016; Foucault 1980, 1981) when discourses become 
hegemonic (Gramsci 1996; Brevini 2020), that they can direct attention from 
the public, construct and promote digital developments, communication 
policy and legitimate modes of governance that would not have been possible 
without the establishment of such a discourse (Brevini and Schlosberg 2016). 
Incomplete discourses that become dominant can shape how society embraces 
technological developments. 


Tech-Determinism, Tech-Solutionism and AI 


The technological deterministic argument that technology can and will fix 
capitalism - and its intrinsic power to exacerbate inequalities of economic, 
racial, gender forms - is far from being a recent elaboration (Gilder 1990; 
Negroponte 1998). To use the words of Mosco, ‘one generation after another 
has renewed the belief that, whatever was said about earlier technologies, the 
latest one will fulfil a radical and revolutionary promise’ (Mosco 2004, 21; 
Brevini 2020). Mosco (2004) rightly reminds us of James Carey’s (1992) work 
that discussed how machines have often been framed employing a powerful 
religious ethos: ‘in contemporary popular commentary and even in technical 
discussions of new communications technology, the historic religious under- 
current has never been eliminated from our thought (Carey 1992, 18). 

As a result, technology becomes the most powerful weapon purporting to lift 
the global capitalist system out of its recurrent crises; and virtually any social 
problem can be subject to a technical and technological fix (Kurzweil 1985). 
Development of digital technology, we are reassured, will empower people out 
of radical inequalities, while naturalising market-based solutions to every issue 
of governance. Raymond Williams, one of the most established cultural theorists 
to come out of Britain, offers a fruitful definition of technological determinism 
as a ‘largely orthodox view of the nature of social change’ (Williams 1974, 13). 
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Furthermore, he explains: “The basic assumption of technological determinism 
is that a new technology - a printing press or a communications satellite - 
“emerges” from technical study and experiment. It then changes the society 
or sector into which it has “emerged” (Williams 1985, 129). On the contrary, 
despite William’s belief in the opportunities offered by innovation, he held that 
‘technology is always in a full sense social, thus its development and usage are 
always shaped by the social relations of the society in which they are adopted 
(Williams 1981, 227). 

Williams was writing at the time when, by the late 1970s, the so called ‘infor- 
mation revolution’ was just emerging as the new dogma in government and cor- 
porate planning (Dyer-Witheford 1999). But the information revolution myth 
kept getting stronger throughout the 1970s, 1980s and 1990s, ‘more attuned 
to the climate of Thatcherism and Reaganism’ (ibid. 21) than to a Keynesian 
state’s framework. This revolution should not come as a surprise since neolib- 
eralism and the information revolution have been endorsed by corporate and 
governments elites as the solution to the ‘growth’ crisis of the 1970s. The neo- 
liberal Clinton administration of the 1990s was an aggressive supporter of the 
technocratic ‘information revolution. In 1994 its congress passed the National 
Information Infrastructure Bill which launched the world famous ‘informa- 
tion superhighway, championed by Al Gore in numerous speeches around the 
world. Another crucial futurologist of the time stressed once again the link 
between technological determinism and neoliberal ideologies. Francis Fukuy- 
ama’s influential book The End of History (1992), proclaimed that the end of the 
Cold War demonstrated the collapse of any reasonable alternative to neoliber- 
alism. Moreover, in order to reinstate the alliance between neoliberalism and 
technology, in The Great Disruption (2017) Fukuyama argues: 


A society built around information tends to produce more of the two 
things people value most in a modern democracy - freedom and equal- 
ity. Freedom of choice has exploded, in everything from cable channels 
to low-cost shopping outlets to friends met on the Internet. Hierar- 
chies of all sorts, political and corporate, have come under pressure and 
begun to crumble. (Fukuyama 2017, 4) 


In sum, this hegemonic Silicon Valley discourse reaffirms again and again that 
technological progress not only provides newly enhanced individual freedoms 
but will lead to radical social change. 

As a consequence, what has been dubbed as technological solutionism 
becomes the only logical consequence of late capitalism (Levina and Hasinoff 
2016). The term tech solutionism has been popularised by Evgeny Morozov in 
his 2013 book To Save Everything, Click Here as: 


Recasting all complex social situations either as neatly defined problems 
with definite, computable solutions or as transparent and self-evident 
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processes that can be easily optimized - if only the right algorithms are 
in place! - this quest is likely to have unexpected consequences that 
could eventually cause more damage than the problems they seek to 
address. I call the ideology that legitimises and sanctions such aspira- 
tions ‘solutionism. (Morozov 2013, 5) 


From its beginnings in the 1950s, AI has not been exempted from these claims 
of offering a ‘solution’ to the inequalities of capitalism (Brevini 2020; Natale and 
Ballatore 2020; Elish and Boyd 2018). On the contrary, it has been surrounded 
by evocative claims about the imminent creation of a machine capable of sur- 
passing the potentials of humankind. AI has often been hailed as the magic tool 
to rescue the global capitalist system from its dramatic failures (Brevini 2020). 

Recent studies on popular and public debates on AI have started to show the 
extent ofthe dominance ofthis tech-deterministic ideology, especially in the US 
(Mayer-Schönberger and Cukier 2013). For example, Elish and Boyd’s research 
(2018) on Al rhetoric, concluded that ‘through the manufacturing of hype and 
promise, the business community has helped produce a rhetoric around these 
technologies that extends far past the current methodological capabilities’ 
(Elish and Boyd 2018, 58). In exploring public discourse shaping the popular 
imagination around possible AI futures, Goode (2018) observes that contem- 
porary discourse is: 


skewed heavily towards specific voices - predominantly male science fic- 
tion authors and techno-centric scientists, futurists and entrepreneurs - 
and the field of AI and robotics is all too easily presented as a kind of 
sublime spectacle of inevitability (...) that does little to offer lay citizens 
the sense that they can be actively involved in shaping its future. (Goode 
2018, 204) 


Furthermore, the latest study on media coverage of Alin the UK conducted by 
the Reuters Institute (Brennen, Howard and Nielsen 2018) showed that the UK 
media coverage of AI was overwhelmingly influenced by industry concerns, 
products and initiatives. 

Thus, this chapter aims to contribute to these scholarly debates by investi- 
gating hegemonic discourses about AI emerging from the European Unions 
official strategy on AI. In particular, it will highlight the most crucial myths on 
which hegemonic discourse is based. 


Developing AI in Europe 
The journey to develop the AI strategy in the EU started in 2018, when the 


European Commission presented the ‘Declaration of Cooperation on AT 
now signed by all 28 Member States, including Norway. In the Declaration, 
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Member States agree to a continuous dialogue to work together towards 
‘a comprehensive and integrated European approach on AI and, where 
needed, review and modernise national policies to ensure that the oppor- 
tunities arising from Al are seized and the emerging challenges addressed’ 
(EU Declaration 2018: 4). 

The AI strategy is developed within the context and legislative packages 
of the Digital Single Market Strategy developed by the EC that include the 
European Data Economy initiatives, the General Data Protection Directive 
and, crucially, the European Cloud Initiative. The latter aims to ‘make it eas- 
ier for researchers, businesses and public services to fully exploit the benefits 
of Big Data by making it possible to move, share and re-use data seamlessly 
across global markets and borders, and among institutions and research disci- 
plines’ (European Cloud initiative 2019). 

On 7 December 2018 the European Commission published a coordinated 
action plan on the development of AI in the EU (European Commission 2018a, 
2018b). It pledged to increase its annual investments in AI by 70% under the 
research and innovation programme Horizon, in order to reach EUR 1.5 billion 
for the period 2018-2020. In its Communication (European Commission 
2018a) the European Commission (EC) reaffirms the belief that ‘AI will help us 
to solve some of the world’s biggest challenges, from treating chronic diseases 
and reducing fatality rates in traffic accidents to fighting climate change and 
anticipating cybersecurity threats (European Commission 20 18a, 2). Therefore, 
the EC put forward a European approach to artificial intelligence based 
on three pillars: 


e connect and strengthen AI research centres across Europe; 

e support the development of an ‘AI-on-demand platform’ that will provide 
access to relevant AI resources in the EU for all users; 

e support the development of AI applications in key sectors (European Com- 
mission 2018b, 1). 


In order to support the development of the AI strategies summarised here, 
the EC established two advisory entities: The High-Level Expert Group on 
AI (HLEG); and the European AI Alliance. The High-Level Expert Group on AI 
is charged with developing proposals for the overall EU’s AI strategy, policy 
and priorities. It comprises 23 members from industry, 19 from academia and 
10 from civil society; and it is further divided into two working groups: one 
on ethics; and one on investment and policy. The second advisory entity, 
the European AI Alliance, is a multi-stakeholder online platform. On the 
platform, EU members can contribute to ongoing discussions on Al, feed- 
ing into the European Commission’s policy-making processes. The Euro- 
pean AI Alliance is conceived as a tool open to all members of society. Cur- 
rently, it is composed of members from civil society, trade unions, companies, 
not-for-profit institutions and consumer organisations. 
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In the first year after its creation in June 2018, the HLEG released two major 
policy documents forming the basis of the latest White Paper on AI, adopted 
in 2020. The first document, Ethics Guidelines on artificial intelligence, put 
forward the concept of “Trustworthy AT’ and the key requirements that AI sys- 
tems should meet in order to be trustworthy (High-Level Expert Group 2019a). 
The second document, Policy and Investment Recommendations (High-Level 
Expert Group 2019b), developed recommendations for AI towards sustainabil- 
ity, growth and competitiveness and inclusion. On 19 February 2020, the Euro- 
pean Commission published a White Paper on artificial intelligence (European 
Commission 2020) aiming to foster a European ecosystem of excellence and 
trust in AI and a report on the safety and liability aspects of AI. The White 
Paper provides a simple, all-encompassing definition of artificial intelligence 
‘Al is a collection of technologies that combine data, algorithms and computing 
power. Advances in computing and the increasing availability of data are there- 
fore key drivers of the current upsurge of AI’ (European Commission 2020, 2). 
The White Paper is clear on twofold goals: on the one hand it aims to support the 
AI uptake and on the other it aims to address the risks linked to particular 
uses of it. These overall aims will be achieved through coordinated measures 
that will streamline research, foster collaboration between member states and 
increase investment into AI development and deployment; and through a 
policy toolkit for a future EU regulatory framework that would determine the 
types of legal requirements that would apply to relevant actors, with a particu- 
lar focus on high-risk applications (European Commission 2020a). Although 
the White Paper does not set out a concrete framework for new Al legislation, 
it does set out the Commission's key priorities. 


Three Myths in Discourses on Al in Europe 


Having outlined the current European Framework developed in the series of 
communications, High-Level Groups reports and lastly, the White Paper on 
AI, this section uncovers the recurrent myths employed in official EU plans to 
develop artificial intelligence. As discussed in the previous section, these myths 
become crucial components of AI discourse, justifying policy-making within 
the European Union. Furthermore, as I will argue, these myths construct a dis- 
course that has the ultimate end of reinforcing the current neoliberal ideology 
of the current stage of capitalism. 


Myth #1: Artificial Intelligence as a Solution for Humanity 
and Capitalism’s Biggest Challenges 


In its communications of 25 April 2018 and 7 December 2018, the European 
Commission set out its vision for AI, which supports ‘ethical, secure and cutting- 
edge AI made in Europe’ (European Commission 2018a). 
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The vision could not highlight in a more striking way how AI becomes the 
solution for humanity’s biggest challenge. The following two paragraphs taken 
from the two official communications (European Commission 2018a) could 
not be clearer: 


AI is helping us to solve some of the world’s biggest challenges: from 
treating chronic diseases or reducing fatality rates in traffic accidents to 
fighting climate change or anticipating cybersecurity threats. (ibid. 2) 


In more evocative terms, the myth of the revolutionary character of Al is rein- 
forced by a comparison with the ‘steam and electricity ‘revolution. 


Like the steam engine or electricity in the past, AI is transforming our 
world, our society and our industry. Growth in computing power, avail- 
ability of data and progress in algorithms have turned Al into one of the 
most strategic technologies of the 21st century. (ibid. 2) 


The High-Level Expert Group on Artificial Intelligence (AI HLEG) goes into 
even greater detail about the capabilities of AI to make humanity ‘flourish, thus 
solving all problems of society. 


We believe that AI has the potential to significantly transform society. AI 
is not an end in itself, but rather a promising means to increase human 
flourishing, thereby enhancing individual and societal well-being and 
the common good, as well as bringing progress and innovation. In par- 
ticular, AI systems can help to facilitate the achievement of the UN’s 
Sustainable Development Goals, such as promoting gender balance 
and tackling climate change, rationalising our use of natural resources, 
enhancing our health, mobility and production processes, and support- 
ing how we monitor progress against sustainability and social cohesion 
indicators. (High-Level Expert Group 2019a, 4) 


Its impossible not to see in this mythical discourse the same rhetoric oftechno- 
crats of the 1990s (Gilder 2000; Fukuyama 1992; Shirky 2008) that argued how 
the new communicative opportunities provided by the internet would enhance 
a new era for democracy (Gilder 2000; Negroponte 1998), the end of history 
(Fukuyama 1992) and the beginning of a new era of freedom. The same ideo- 
logical discourse is replicated in current techno-enthusiast claims about the 
cloud (Nye 1994) more recently debunked by Mosco in his book, To the Cloud: 
Big Data in a Turbulent World (Mosco 2014). 

In pure enlightenment fashion, this absolute faith in technology, embraced 
and supported by cybertarians’ Silicon Valley circles (Dyer-Witheford 1999; 
Brevini 2020) turns into a powerful apology for the status quo and the current 
structure of capitalism, without any real space for critique. 
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Myth #2: Creating Urgency and ‘Preparing Society - Al as Ineluctable 


The second of the most compelling myths emerging from my analysis of EU 
strategies on AI is the myth of ATs perceived ineluctability, built through a 
constant emphasis on its urgency. Consider for example this quote, from the 
European Commission Communication of April 2018: 


The stakes could not be higher. The way we approach AI will define the 
world we live in. Amid fierce global competition, a solid European 
framework is needed. (European Commission 2018a, 2) 


Moreover, the White Paper - that is the latest policy document adopted by the 
EC to establish its framework (European Commission 2020) - stresses again 
the urgency for every sector of Public services to employ AI as soon as possible. 


It is essential that public administrations, hospitals, utility and transport 
services, financial supervisors, and other areas of public interest rapidly 
begin to deploy products and services that rely on AI in their activities. 
(European Commission 2020, 8) 


Overall, discourse stressing the need to hurry up on investments - such as 
‘Europe is behind in private investments on AT (European Commission 2018a, 5), 
or ‘the European industry cannot miss the train (European Commission 
2018a, 5) -are reiterated throughout the documents developing EU strategy on Al. 
So fast paced is the race to adopt AI that the opposite would be inconceivable: 


Without such efforts, the EU risks losing out on the opportunities 
offered by AI, facing a brain-drain and being a consumer of solutions 
developed elsewhere. (European Commission 2018a, 6) 


The myth of AI ineluctability is further enhanced by repetition of sentences 
reaffirming the role of the EU as enabler of AI, with an almost teleological duty 
to ‘better prepare our society for AT (European Commission 2018b, 5) as if its 
divine advent on earth was inevitable. 

This should remind us of the dawn of AI developments in the 1950s (Roszak 
1986), when popular accounts proclaimed the imminent development of intel- 
ligent machines capable of outsmarting the human mind amid promises to 
fundamentally change everything. However, as Goode (2018) recalls, in the last 
decade we have seen a clear increase of predictions that the arrival of superintel- 
ligence is imminent, thus the urgency (Goode 2018) this calls for in producing 
EU level strategies. Claims like “The singularity is near, by Ray Kurzweil, futurist 
and Director of Engineering at Google are indicative of the current ‘anxiety sur- 
rounding the speed with which the technology appears to be developing, some- 
thing that some robotics companies are keen to play up’ (Goode 2018, 199). 
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Unveiling this myth of the ineluctability of AI and its urgency, it is impos- 
sible not to recall Williams’ analysis in Towards 2000 where he stated that 
“The sense of some new technology as inevitable or unstoppable is a product 
of the overt and covert marketing of the relevant interests’ (Williams 1985, 
133). In reality technological development is not predetermined, and alterna- 
tive paths to a market-led development that reinforces the current neoliberal 
status quo are always a possibility (Brevini 2020). 


Myth #3: AI Surpassing Human Intelligence 


Like every institution that developed a strategy for AI, the EC also had to start 
by defining AI. The Communication of the Commission clarifies that: 


Artificial intelligence (AI) refers to systems that display intelligent 
behaviour by analysing their environment and taking actions - with 
some degree of autonomy - to achieve specific goals. AI-based systems 
can be purely software-based, acting in the virtual world (e.g. voice 
assistants, image analysis, software, search engines, speech and face 
recognition systems) or AI can be embedded in hardware devices 
(e.g. advanced robots, autonomous cars, drones or Internet of Things 
applications). (European Commission 2018a, 2) 


Moreover, what emerges from the EU documents is the underlying assumption 
that artificial intelligence will outperform human capabilities. In several docu- 
ments, the EC explains that AI has the capacity to transform ‘our world, our 
‘society, our ‘work’ (Ihe Communication, European Commission 2018b, 1), 
thus implying that its abilities will exceed human cognitive functions. Take for 
example this statement: 


AI needs vast amounts of data to be developed. Machine learning, a type 
of AI, works by identifying patterns in available data and then apply- 
ing the knowledge to new data. The larger a data set, the better AI can 
learn and discover even subtle relations in the data. Once trained, algo- 
rithms can correctly classify objects that they have never seen, in more 
and more cases with accuracies that exceed those of humans. (European 
Commission 2018b, 6) 


In the 1980s Roszak had already implemented the term ‘technological idol- 
atry’ that propagates a deference to computers ‘which human beings have 
never assumed with respect to any other technology of the past’ (Roszak 
1986, 45). This clearly reveals how the construction of AI as machines that 
can outperform human labour helps legitimise current capitalistic structures 
that are indeed capable of generating the technocratic imperative that see the 
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subordination of human labour to computers. Of course, as I have discussed 
above, neoliberalism has long established a privileged relationship with tech- 
nology as the prime mover’ of capitalist growth (Harvey 2005). 


Conclusion 


Ihe AI myths discussed in these pages are very powerful tools for the con- 
struction of a discourse that make us perceive Al as the solution to the major 
problems in our society, including the inequalities brought about by capital- 
ism and other major crises such as climate change and global health emergen- 
cies. Through these myths, AI then becomes the technological saviour, whose 
advent is ineluctable. As such, when the artificial machine arrives - in this 
future/present which is always inevitably imminent - it will manifest as a supe- 
rior intelligence to solve the problems that capital economies have themselves 
created. Eventually, AI will outsmart humans to mend that damage and amelio- 
rate further risks that capitalism inevitably occasions. 

The recurrent myths that are omnipresent in the European Framework for 
AI have two major consequences. Firstly, they structure a hegemonic dis- 
course that makes it impossible to think of alternative paths, framing resist- 
ance as futile because technological development is predestined. Accordingly, 
they legitimise a neoliberal ideology that pushes consumerism and produc- 
tivity above all values and strips technology from the social relations that are 
at the basis of technology development (Williams 1985; Brevini 2020). Sec- 
ondly, they redirect public discourse, by obfuscating and inhibiting a serious 
debate on the structural foundations of AI, its progressively concentrated own- 
ership and the materiality of its infrastructures. Taken together, these myths 
of AI, construct a type of discourse that frames the problem of AI in a way 
that excludes any emphasis on crucial questions of ownership, control and the 
public interest. It also diverts attention from known problems of inequality, 
discrimination and bias of data analysed algorithmically that lies at the heart 
AI systems (Brevini and Pasquale 2020). When these crucial questions are 
asked, they are only addressed through the AI ethical’ framework that has 
little to say about the structural inequalities on which Al is built (Wagner 2018). 

This optimism for AI possibilities and achievements so popular in Europe 
and in the West, is obviously fuelled by extremely effective lobbying efforts 
by the most powerful technology giants that are already dominating the 
market and debate. From Alphabet to Amazon, to Microsoft, IBM and Intel, 
we have evidence that the giants of Silicon Valley are investing billions both 
on AI developments and on setting the terms of public debates on AI and 
determining policy outcomes (Benkler 2019). Thus, a central concern of this 
chapter is the migration of strategic decisions and choices on the direction of 
AI development from government to corporate board rooms: the privatisation 
of public policy. 
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Major lobby groups go in to bat for their vested interests in the policy arena, 
armed with funded academic research on the benefits of Al and efficiency. For 
example, a report published in 2019 by the New Statesman revealed that in five 
years Google has spent millions of pounds funding research at British universi- 
ties including the Oxford Internet Institute (Williams 2019), while DeepMind, 
Alphabet’s own AI company, has specifically supported studies on the ethics of 
AI and automated decision-making. Correspondingly, Facebook donated US 
$7.5m to the Technical University of Munich, to fund new AI ethics research 
centres. Another troubling case is the US-based National Science Foundation 
program for research into ‘Fairness in Artificial Intelligence, co-funded by 
Amazon (Benkler 2019). As scholar Yochai Benkler explained, the digital giant 
has ‘the technical, the contractual, technical and organizational means to pro- 
mote the projects that suit its goals’ (ibid. 2019). Hence, ‘Industry has mobilized 
to shape the science, morality and laws of Artificial Intelligence’ (ibid. 2019). 

Moreover, this portrayal of AI as the magic, divine hand that will rescue 
society also obfuscates the materiality of the infrastructures that are central to 
the environmental question that has been so consistently and artfully ignored 
(Brevini 2020). AI relies on technology, machines and infrastructures that 
deplete scarce resources in their production, consumption and disposal, thus 
increasing amounts of energy in their use, and exacerbating problems of waste 
and pollution. AI generates an array of environmental problems, most nota- 
bly energy consumption and emissions, material toxicity and electronic waste 
(Brevini and Murdock 2017). Yet these myths help build a discourse that is 
skewed heavily towards specific voices - predominantly corporate and neolib- 
eral - that build a so-called common sense that is too pervasive to challenge. 
AI brings us to a present/future in which alternative paths to current capitalism 
are unthinkable. And so, we surrender to our inevitable destiny of a new world 
order of wellbeing brought by AI, shaping that future for the benefit of the most 
powerful who built its technology and framed its hegemonic discourse. 
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CHAPTER I0 


AI Bugs and Failures: How and Why 
to Render AI-Algorithms More Human? 


Alkim Almila Akdag Salah 


Introduction 


When we look at the history of computer art, we see many instances where the 
artworks were created rather by ‘accident’ than by carefully worked out pro- 
cesses and deliberately written codes. For instance, one of the first computer 
artists, Michael Noll, acknowledged that the idea to experiment with comput- 
ers to achieve artistic patterns came to him after a programming error, which 
resulted in an interesting graphical output (Noll 1994). The most eye-catching 
example of this sort comes from an entry for the exhibition Cybernetic Ser- 
endipity: ‘A bug in the programme [sic] effectively randomized the text given 
to it ... but we are not sure as we failed to make the programme do it again. 
At any rate, this “poem” is all computer’s own work? (McKinnon Wood and 
Masterman 1968, 54). During those early years of computer art, there was an 
apparent tendency to ‘anthropomorphise computers, and the split second of 
humanity that a bug bestows on the computer was maybe the best chance to 
reach that aim. A computer graphics competition held in 1968 by California 
Computer Products is cited to publish a statement ‘that they were convinced 
that computer art would be accepted as a recognized art form ... because it gives 
a humanizing aura to machinery (Reichardt 1971, 74). 
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Today, thanks to the rise of big data, computing power and mathematical 
advancements, and the introduction of convolutional neural networks (CNNs), 
we live with intelligent algorithms (i.e. weak AI), in many aspects of life." For 
example, the effects of these algorithms in digital visual production covers rec- 
ommendation systems, automatic image editing, analysing and even creating 
new images, but these are not recognised as ‘intelligent’ systems (Manovich 
2020). What fascinates the human mind are still the observances of failures. 
A prominent example is the images and videos created with the Deep Dream 
algorithm, which was originally devised to unearth what lies in the hidden lay- 
ers of CNNs to capture the workings and failures of the system (Simonyan and 
Zisserman 2014). These images are hailed by some as artworks on their own 
(Miller 2019). 

Autonomous AI systems such as self-driving cars, or autonomous lethal 
weapons are expected to work in a framework called ‘explainable AT, under 
meaningful human control, and preferably in a fail-proof way (Santoni de Sio 
and Van den Hoven 2018). Here, I will discuss case studies where the opposite 
framework will prove more beneficial, i.e., in certain contexts, such as cultural 
and artistic production or social robotics, AI systems might be considered more 
humanlike if they deliberately take on human traits: to be able to err, to bluff, to 
joke, to hesitate, to be whimsical, unreliable, unpredictable and above all to be 
creative. In order to uncover why we need ‘humanlike’ traits - especially bugs 
and failures - I will also visit the representations of the intelligent machines in 
the imagination of popular culture, and discuss the deeply ingrained fear of the 
machine as the ‘other’ 

The aim of the chapter is twofold: first, by reiterating the history of com- 
puter art and comparing it to how artistic production in/with AI is used and 
interpreted today, I pinpoint how the discourse of artistic (computational) pro- 
duction has changed. Second, by visiting classical definitions of AI and jux- 
taposing them with the public expectations and fears, I will uncover how the 
myths about AI are assessed when it is tasked to take on not only human jobs, 
but human traits as well. 

In this chapter, I will build our framework around the famous discussions of 
the Turing test, the Chinese Room and what it means to have a computational 
system for creativity and arts. I will then look at the history of computer art by 
assessing the early artworks, exhibitions as well as magazines devoted to the 
genre. Especially the latter gives us insight into what the experts’ expectations 
of computers were. I will furthermore delve into the history of sci-fi and build 
bridges between these early artworks and sci-fi novels and movies of the time 
to understand the reaction of the public to the idea of intelligent/sensuous (i.e. 
human like) machines. Moving to today, I will visit two artists working within 
the framework of AI and Big Data, who proposed two extreme approaches to 
this framework: Refik Anadol, who enlists Big Data and AI in a black-box fash- 
ion for generating big displays of contemporary aesthetics, and Bager Akbay, 
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who reveals the working of AI by generating instances of occurrences between 
the audience and the AI. 


A Useful Framework for AI or the Ghost in the Machine? 


Some sixty years ago, a part of the computer science community embarked 
upon an ambitious research program named artificial intelligence. Summar- 
ily, the task at hand was to write an intelligent computer program; one that 
could simulate human thinking, and while at it, why not, properly think and 
even be conscious, just like a human. They had just realised that computers 
were able to handle arithmetical and logical operations much better than an 
ordinary human being, and a whole wide world of opportunities opened up 
before them. But after some initial effort, the researchers saw two things: the 
aim of implementing intelligence was an ill-posed problem, because there 
was no satisfactory definition of intelligence. The concept of intelligence, just 
like many defining characteristic of human beings, is normative and vague. 
The second realisation would come a little later, as it required more failures: the 
internal dynamics of many human endeavours were unknown, and misjudged. 
People thought that understanding and speaking was easy, whereas playing 
chess was difficult. Thus, chess was seen as a benchmark of intelligence. Years 
later, when a computer program, Deep Blue (Hsu 2002), was able to beat the 
world champion of chess, cognitive science had already contributed much 
to our understanding; nobody claimed that the computer was intelligent, let 
alone conscious. 

The first and foremost effort to frame intelligence came from one of the great- 
est minds of the era. Alan Turing, in his classical essay ‘Computing Machinery 
and Intelligence’ proposed an imitation game, where a computer chats with a 
person and passes the game if it can keep up the appearance (of being human) 
for about five minutes (Turing 1950). Turing’s expectations of the future of 
computers was quite close to the mark: 


The original question, ‘Can machines think?’ I believe to be too mean- 
ingless to deserve discussion. Nevertheless I believe that at the end of 
the century the use of words and general educated opinion will have 
altered so much that one will be able to speak of machines thinking 
without expecting to be contradicted. (Turing 1950, 442) 


In 1980, John Searle wrote the most controversial critique of artificial intel- 
ligence. His famous Chinese room argument is as follows: suppose there is a 
room with two small slots, and a person within it. One of the slots is used to 
pass this person little pieces of paper with Chinese characters on it. There is 
also a rulebook in the room that tells the person inside what kind of symbols 
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he should use in order to respond to the incoming ‘squiggles. The response 
‘squoggles’ are put through the second slot. Although the symbol exchange 
can be seen as a perfectly normal conversation for a person who understands 
Chinese, the person in the room does not know Chinese. Having consciousness 
and simulating one are different for Searle, and no machine will ever think, it 
can just simulate, or act as if it is thinking (Searle 1980, 355). 

The responses to Searle are numerous (Harnad 1989), and the debate con- 
tinued for about ten years. What interests me here is not the debate itself, but 
rather its emphasis on intentionality. For philosophers of language, what is 
meant by intentionality is largely an issue of how symbols can have meaning. 
Searle argues that a computer simulation, no matter how good it is, will never 
project more than its programmer's intentions, i.e. a computer program can 
never have intentions (and mental states), because it is written according to 
some syntactical rules, and it lacks the connections to semantic access. If such 
written programs can fool us into believing that they are intelligent beings, 
this does not prove that these programs are operating on the semantic level; it 
only shows that we are deceived by the programmer's ghost, which acts like a 
remote-control system through the program. 

At the end, Searle’s approach comes down to a simple point: an inorganic 
entity cannot develop intentional states and cannot become conscious. 
Douglas Hofstadter asserts that Searle’s argument comes from a dualist point 
of view, which is denied fiercely by Searle in his reply to Hofstadter. I think that 
there is some truth in Hofstadter’s claim; after all, a search for a human soul in 
a digital computer, even if it is run under the name as ‘intentionality/conscious- 
ness’ is suggestive enough for a belief in body/mind distinction. Furthermore, 
Daniel Dennett (1982) points out that the furious defenders of the Chinese 
Room argument are known dualists, and the main critics of Searles argument 
are de-emphasising the importance of consciousness (even) in humans. 

There is the unmistakable Cartesian ego that Dennett and many others see 
in Searles argument, the ghost that Gilbert Ryle wanted to abolish, the implicit 
belief of the superiority of the human that is the hallmark of the modern era. 
In short, Turing tries to move the definition of intelligent machines outside the 
realm of human traits (we do not need to measure how intelligent a machine 
is, we only measure how well it fits within everyday relations with a human). 
On the other hand, Searle tries to kill the concept of intelligent machines by 
comparing not the ‘behaviour’ of these machines to human behaviour, but their 
‘nature’ to that of human nature. 

Today, AI is more and more associated with words that are reserved for 
humans: autonomy, learning and interpretation. For example, Haenlein and 
Kaplan (2019) state that AI is commonly defined as ‘a system’s ability to inter- 
pret external data correctly, to learn from such data, and to use those learnings 
to achieve specific goals and tasks through flexible adaptation. Rahwan et al. 
(2019) called for a new science on machine behaviour as a field that ‘is con- 
cerned with the scientific study of intelligent machines (i.e. virtual or embodied 


Al Bugs and Failures 165 


AI agents), not as engineering artefacts, but as a class of actors with particu- 
lar behavioural patterns and ecology. A computer program that can learn 
goes beyond what it is programmed to do. The developments in the field (and 
the transformation of the definitions of AI) notwithstanding, the position of 
Turing, as well as that of Searle, are not totally overturned yet. Searle’s vague 
definition of Strong AI (to create intentionality artificially) has led to the term 
‘weak AT; and both terms are still in use today (Morgan 2018). Weak AI deline- 
ates an artificial agent that succeeds to reach a goal in a given environment by 
computing a function from its sensory inputs to its actions. From chatbots we 
encounter on websites to a washing machine that can use sensors to calculate 
the washing load and adjust the water usage accordingly, all ‘smart’ applications 
and tools fall under weak AI. Strong AI, or artificial general intelligence (AGI), 
on the other hand, points out to a more human-like intelligence that is flexible 
enough to learn abstractions of any novel domain relatively quickly, and per- 
form with increasing accuracy on this domain.’ 

Here I would like to draw attention to a recent discussion of strong ver- 
sus weak computational creativity using AI techniques to visualise ‘tracing’ 
line drawings (Al-Rifaie and Bishop 2015). This work invites its readers to a 
Gedankenexperiment similar to the Chinese Room, where the program gener- 
ates drawings as outcomes, and for the audience outside the Creativity Room, 
these drawings are genuine artistic productions. The authors’ arguments are 
similar to those of Searle, and they conclude that it is not possible to create 
a creative machine in the general sense. However, they transfer the idea of 
strong/weak AI to machine creativity: An analogy could be drawn to computa- 
tional creativity, extending the notion of weak AI to ‘weak computational crea- 
tivity, which does not go beyond exploring the simulation of human creativity; 
emphasising that genuine autonomy and genuine understanding are not the 
main issues in conceptualising weak computationally creative systems. Con- 
versely in ‘strong computational creativity, the expectation is that the machine 
should be autonomous, creative, have ‘genuine understanding’ and other cog- 
nitive states’ (ibid.). This differentiation, as we will see, is helpful in assessing 
both computer artworks and Al-arts, as well as the intentions and goals of the 
programmer or the artist. 


A Useful History: A Look at the First Computer Artworks 


For many, the first computer art contest held by the journal Computers & People 
in 1963 marks the beginning of computer art. During the year 1965, similar 
exhibitions were hosted on both sides of the Atlantic. In February, the Stuttgart 
gallery of Wendelin Niedlich greeted the art world with the exhibition Gen- 
erative Computergrafik. On display were the first computer artworks by Georg 
Nees. Following this exhibition, Georg Nees took part in another exhibition, 
Computergrafik, on 5 November, this time with his co-scientist Frieder Nake. 
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Their works were on display for the entire month of November. Meanwhile, 
Michael A. Noll and Bela Julesz from Bell Laboratories were asked to show 
their works on the other side of the ocean, and the first computer art exhibi- 
tion of America came to life in Howard Wise’s New York gallery between 6-24 
April. This exhibition had a simple and descriptive name, just like its forerun- 
ner; it was called Computer Generated Pictures.’ 

The driving force behind the German wing of computer art was Max Bense, 
who was an influential aesthetician and a well-known figure among art cir- 
cles. He was greatly interested in cybernetics and information theory, and his 
attempts to make use of these concepts in art theory gave the impetus and the 
right climate for scientists like Nees and Nake to publish their works as art. 
Bense’s objective was to use computer generated pictures for finding ‘quan- 
titative measures of the aesthetics of objects’ (Candy and Edmonds 2002, 8). 
Soon after the first exhibitions, Nees became a student of Bense, and wrote his 
dissertation entitled ‘Generative Computer Graphics. The thesis most notably 
included several highly principled computer programs to generate graphical 
output, based on Bense’s ideas as outlined in his book Aesthetica (Nees 1969; 
Bense 1965). 

Computer art relied on two mechanisms in the beginning: the artist’s inten- 
tions, and the use of computer, respectively. During the early years of the move- 
ment, computers were found only in research centres and could not be operated 
by a single user, thus naturally implying coaction between the artist and the 
scientist. Hence, computer art, by definition, has a hybrid character, combining 
elements of art and technology, oscillating between working in the constraints 
of scientific agenda and creating art products. The investment needed to digest/ 
understand latest scientific research and apply these ideas, methods and tools 
into arts asks for a new type of artist/scientist (well-versed in both discourses), 
or teamwork consisting of scientists and artists who have developed a hybrid 
language to overcome the problem of illiteracy that arises due to the lack of a 
common terminology. The early computer artworks were produced by scien- 
tists who were both interested in pushing the abilities of the computers, as well 
as searching for quantifiable means of aesthetic production and experience. 

Although Georg Nees actively sought procedural descriptions for basic 
principles of aesthetics, most of the first artworks were created rather by ‘acci- 
dent’ than by carefully thought processes and deliberately written codes. For 
instance, Frieder Nake’s first art objects were the results of a test run for a newly 
designed drafting machine. In a similar vein, Michael Noll acknowledges that 
the idea to experiment with computers to achieve artistic patterns came to him 
after a programming error, which resulted in an interesting graphical output: 
‘Lines went every which way all over his plots. We joked about the abstract 
computer art that he had inadvertently generated’ (Noll 1994, 39). 

A majority of these first artworks of the movement came either from research 
laboratories or universities, and their accidental nature is highly relevant in 
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understanding the genesis of the movement. Noise, errors (or bugs as they are 
called in computer engineering jargon) and accidents were an unavoidable part 
of these experiments and Noll was by no means the only one getting excited 
about tracing the outcomes of such bugs to generate artworks. An ‘erring’ 
computer - if it provides an appealing output - looks like acting on free will, 
running against the orders by providing such an unanticipated result. The com- 
puter of course does not have a free will, and it was usually not unpredictable 
at all. It did not go beyond the programmed code, but it could get input 
from the world, and use its uncertainty to drive its own unpredictability. How- 
ever, the unexpected result was mostly the outcome of an error in the code. 
But when the expectations of the users are not answered due to such a bug, and 
especially when the users are novices (like most of the computer artists were), 
and therefore have no clue of what the error in the code might be, the impact 
of the bug is unavoidable: the computer becomes human in the eyes of the 
novice. The perception of computers in popular culture, i.e. in the eye of 
the novice and the layman, is best read in the science fiction novels produced 
during the initial years of computer science. To see the expectations of the 
experts (in arts, as well as in sciences) a summary of publications on intelligent 
and spiritual machines suffices. 


Intelligent or Spiritual Machines: 
Which One is More to be Feared? 


Leonardo, a journal devoted to the intersection of art and sciences, published 
various papers on the issue of intelligent machines and their relation to art. 
Especially the early papers show a high degree of expectancy and a belief in the 
unlimited potential of computers. Michael Thompson declares in 1974 that ‘in 
order to be of value to artists, computers must be perceptive and knowledgeable 
in visual matters. Being “perceptive” means that they should be programmed to 
deal with phenomena that artists perceive and find interesting. Being “knowl- 
edgeable” means here that the computer can use information stored in it to take 
appropriate courses of action (Thompson 1974). In 1977, Michael Apter raised 
the stakes higher by conceptualising a computer that develops an aesthetic taste 
(Apter 1977). Even during the so-called AI winter, i.e. when the expectations 
of AI research were not met and the funding for AI plummeted, Marthur Elton 
claimed that we can build creative machines that will allow us to understand 
‘ourselves and machines’ better (Elton 1995, 207). Robert Mueller elaborated 
that once realised, such devices will ‘mark the death of the personal human 
imagination. He nonetheless concluded that creative machines could pave 
the way to new art venues (Mueller 1990). The most preposterous claim put 
forward around that time came from McLaughlin’s (1984) prediction that in 
100 years intelligent machines will dominate the earth. 
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There was an opposite view against this belief in computers’ abilities to 
develop intelligence and creative abilities in the (near) future. A great many saw 
computers only as ‘symbol processors, i.e. machines that are a little better than 
calculators. Both sides had little understanding of what computers really were, 
and what could be expected from them. Harold Cohen, who is famously known 
for his artificial painter program AARON,” observes this diversification in the 
audience of his exhibitions: “The public seemed to be divided by pretty evenly 
between un-sceptical believers and unbelieving sceptics. The believers were 
happy to believe that computers could do anything and consequently accepted 
the idea, with neither difficulty nor understanding that mine was doing art. 
The sceptics thought computers were just complicated adding machines and 
consequently, experienced insurmountable difficulty and equally little under- 
standing, in believing that mine was doing what I said it was doing’ (Cohen 
2002, 97). 

The public opinion oscillated between these two ends; both of which were 
equally dangerous since both were open to wild speculations or predictions 
about the future of computers and their role in the society. The science fic- 
tion novels of those times are full of telling examples about this oscillation. In 
many science fiction novels, the computer is depicted either as a giant machine 
controlling the human society, in a sense replacing the government (this is the 
exaggerated version of the belief that computers were symbol processors) or 
as a substitute for a human, where the computer or the robot takes on specific 
roles like the teacher, police, surgeon, adviser, etc. (following on the belief that 
computers could do anything). In both instances, the computer is portrayed as 
superior to humans, and the only way to make humans triumph over comput- 
ers is to overemphasise certain human traits. 

For example, in “Ihe God Machine’ (Caiden 1989), the supercomputer col- 
lapses because it cannot bluff as the human opposing it; in both ‘Variable Mar’ 
(Dick 1957) and ‘Fool’s Mate’ (Sheckley 2009) the computer is defeated because 
it cannot predict human actions; or in “The Moon is a Harsh Mistress’ (Heinlein 
1966) the super machine cannot understand why a joke is funny. The SF lit- 
erature is full of these examples, but the one example that is most relevant to 
the topic of the present work is the one where the computer (or the robot) 
develops beyond being a calculating machine and gains a very peculiar human 
ability: creativity. 

One of Asimov’s best stories, “The Bicentennial Man’ is based on this idea 
(Asimov 2000). The hero of the story is one of the earlier robots crafted for 
general usage and sold into a most wealthy household. When it develops the 
ability to make art-pieces, the producing company acknowledges this as a 
defect, and offers to replace the robot. Its owner decides to keep this peculiar 
robot, and gives it the privilege to earn money through selling its works. As the 
story unfolds, the robot becomes more and more human and demands to have 
more rights; first it fights for its freedom, then it asks to be called a human. 
However, the price for humanity is very high. It is not enough to be creative, to 
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have the wish and need for freedom, or the longing for humanity; it is not even 
enough to look and act like a human. The price for humanity is the humble 
attitude of giving up all the superior abilities; and in this case, the most supe- 
rior (and dangerous) faculty of the robot is its immortality. Thus, in order to 
become human, it has to accept death. 

With every attempt to move computers into the territory of human intel- 
ligence, the definition of intelligence or the understanding of human abili- 
ties changes. In the literature of AI history, this dilemma is called as the 
AI Effect’ (Haenlein and Kaplan 2019). However, since the source of the prob- 
lem is rather in the disinclination of humans to accept the capabilities of com- 
puters in taking on faculties that are attributable to humans only, even the 
Turing test which was devised to avoid this problem, cannot offer a tangible 
solution. At the bottom of this disinclination lies the narration of humans as 
superior beings in the universe. This belief, which has religious roots, shapes 
the world view of its adherents in such a way that there is no place for comput- 
ers beating down humans in logical operations, let alone in more delicate traits 
like writing poetry, or making art. We should not forget that the definition and 
understanding of human intelligence has been shaped by Al research as well, 
and there are ample examples in science fiction to accommodate the discourse 
around posthumanism (Hayles 2008) and transhumanism movements. 


A Provocative Experiment 


The fear of computers, the fear of intelligence in an ‘other’ that is capable of 
thinking and creating, played a role in forming a certain reluctance to associate 
any kind of art with computers in the public mind. Obviously, there were other 
problems, most importantly the fact that a normative definition of art involves 
the intentions of the artist. Computer art as a genre followed this normative 
definition, and put emphasis on the intentions of the artist/scientists. There- 
fore, intentionality, or the lack thereof, was a much more relevant issue. 

During 1960s, the art world was discussing the relevance of chance occur- 
rences in creating art. The idea of randomness as opposed to intentions has 
surfaced now and again throughout many 20th century art movements and 
made quite an impact. Surrealists tried to let their subconsciousness take 
over by giving up their self-control over their minds. A similar approach - 
albeit with different reasons and results - was followed by Dada artists dur- 
ing the 1920s. During the late 1960s randomness and chance were important 
factors in the artworks of major figures like John Cage. As the computers 
entered the stage, they offered an easier way to explore these chance occur- 
rences. In an article published in 1968 about art and technology, Douglas 
Davies particularly emphasises the effect of chance and its role in the history 
of aesthetics, as well as its immediate relation with technology and control 
(Davies 1968). 
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Random occurrences, or the ability to create randomness in an artwork was 
seen as one of the advantages offered by the computer. This is one of the most 
debated topics of early computer art. Reichard, as many others, refutes the idea 
of putting computer-generated randomness on a par with the chance occur- 
rences sought by action painters like Alan Davie. As an example, Reichardt 
refers to one of Alan Davie’s paintings where the words cat and mouse are 
added to the painting, because a cat entered the room and walked over the 
painting while Davie was working on the painting by pouring paint onto 
the canvas. Reichardt is of the idea that such an occurrence cannot be dupli- 
cated or mimicked by computers (Reichardt 1971). However, according to Max 
Bense’s theory of Generative Aesthetics, ‘randomness involved in computer 
graphics replaces that aspect in art which is described as intuitive’ through 
computer procedures. “Thus the randomizing procedures in computer technol- 
ogy are analogous to an artist’s intuition’ (Davies 1968, 8). 

When Georg Nees displayed his randomly distributed geometric shapes in 
an art context, the reaction of the art community in Stuttgart was quite fierce: 
‘Some of them (the artists) became nervous, hostile, furious. Some left. If 
the pictures were done by use of a computer, how could they possibly be art? The 
idea was ridiculous! Where was the inspiration, the intuition, the creative act? 
What the heck could be the message of these pictures? They were nothing but 
black straight lines on white paper, combined into simple geometric shapes. 
Variations, combinatorics, randomness ... but even randomness, the artists 
learned, was not really random but only calculated pseudo-randomness, the 
type of randomness possible on a digital computer. A fake, from start to end, 
christened as art!’ (Nake cited in Candy and Edmonds 2002, 6). Many artists 
were not ready to accept a randomness created by computers for real. Actually, 
the fact that these works were showcased as art was not as puzzling and discon- 
certing as the realisation that the same works could really be passed as made 
by human hands. 

In his book AI Aesthetics (2020), Manovich compares early computer art with 
Al-arts, and makes an interesting observation. The early computer artworks are 
abstract in nature, not related to human affairs except the concern on aesthet- 
ics, whereas today, we see more and more works that mimic many layers of arts. 
Manovich furthermore proposes three ways to define AI art. The first proposi- 
tion comes from designing a “Turing Al arts’ test, the other two definitions are 
asked to not only mimic existing art in a convincing way, but to go beyond the 
cultural production of today, and generate truly innovative products. The defi- 
nitions here differ only how they achieve these innovations. But if we return to 
a Turing Al-test that follows the conditions ‘if art historians mistake objects a 
computer creates after training for the original artifacts from some period, and 
if these objects are not simply slightly modified copies of existing artifacts, such 
computer passed “Turing Al arts” test ... In this definition, art created by an 
AI is something that professionals recognize as valid historical art or contem- 
porary art? 
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Actually, history of computer art already witnessed such a Turing Al arts 
test. The artworks generated by the computer back then were not ‘sufficiently’ 
different than original artworks, but considering the time frame when the 
experiments were run, this might be excused. Michael Noll wrote a program 
to simulate Mondrian’s ‘Composition with Lines, which is a black-white com- 
position. The end result ‘Computer Composition with Lines’ was quite similar 
to the original, and Noll used it in an experiment performed with 100 subjects. 
In the experiment, the subjects were shown Mondrian’s and Noll’s composi- 
tions, and subsequently were asked to about the authenticity of the pictures 
shown; the subjects had to identify Mondrian’s picture, and they were asked to 
give explanations on why they chose one picture over the other. The third page 
was called the ‘preference’ questionnaire, asked the subjects which picture they 
liked more, encouraging them to give specific reasons for the preference. Noll 
published the results as an article in The Psychological Record in 1966, and this 
article is reprinted in various edited collections since then (Noll 1966). 

In his paper, Noll carefully noted his methods for designing the experiment 
to explicate how a control group was formed to see whether his subjects were 
prejudiced against computer art. To prevent any such prejudice, the control 
group was first asked to choose a painting, and only then expected to identify 
the Mondrian painting. The statistical analysis showed that the order of ques- 
tions did not have any significant bearing on the preference of the subjects. 
The results were quite ‘thought-provoking’ as Noll noted in his paper. Of the 
subjects, 59% preferred the computer-generated image. Moreover, only 28% 
of the subjects were able to identify Mondrian’s painting correctly, and most 
of them had a ‘technical’ background. Noll’s explanation for the higher correct 
identification rate by subjects with technical backgrounds was that they were 
familiar with computer programming and had an advantage at guessing which 
picture was generated by a computer. On the other hand, Noll was also con- 
vinced that Mondrian’s painting was carefully planned and conducted accord- 
ing to an algorithm, which he himself was unable to discover. In comparison 
with this calculated painting, Nolls computer design struck the eye as being 
more ‘random. Consequently, a higher percentage of the subjects preferred the 
computer-generated image. Noll concluded that randomness creates a feeling 
of creativity, and especially for the non-technical subjects that was equivalent 
to an indication that a human crafted the painting. 

Noll’s explanation to the rather astonishing preference for the computer- 
generated image was that all the subjects were familiar with computer technol- 
ogy and were using it in their everyday lives.’ The subjects were recruited from 
his own work environment, and thus represent a biased sample. It was quite 
unnatural in 1965 to have so many subjects familiar with computers, as the 
majority of the population had not even encountered a computer in their lives. 
According to Noll, the subjects did not have any prejudice against using com- 
puters for creating art as a result of this familiarity. He further commented 
that the results may have been quite different if the subjects had been coming 
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from an artistic background, anticipating a negative reaction against computer 
usage in arts. Although later he also did an experiment on subjects with artistic 
backgrounds, the experimental setup was quite different, and does not lead toa 
direct comparison (Noll 1972). 

Nolls experiment is particularly relevant, because it demonstrated that 
a computer-generated picture could be confused with or even preferred over a 
human-generated artwork. Through the experiment, he challenges the possi- 
bility of reading the artistic intentions behind an artwork (and questions one of 
the basic assumptions of art history as a discipline): “The experiment compared 
the results of an intellectual, non-emotional endeavour involving a computer 
with the pattern produced by a painter whose work has been characterised as 
expressing the emotions and mysticism of its author. The results of this experi- 
ment would seem to raise some doubts about the importance of the artist’s 
milieu and emotional behaviour in communicating through the art object’ 
(Noll 1966, 10). If the artwork cannot mirror its creators’ intentions, thoughts 
and ideas, can we still claim that the artwork reflects its era, more than any 
everyday object? 


Where Are We Now: Computer Art, Aesthetics and AI Art 


Within computer vision and multimedia retrieval, computer-based analysis 
of artworks has received increasing attention in the last two decades (Spratt 
and Elgammal 2014). The research focused on creating automatic programs 
that, given an artwork, can identify the artist, the style or the production date, 
as well as search for stylistically similar artworks in a collection (Stork 2009). 
While some of this research followed reductionist perspectives and was heav- 
ily criticized for losing sight of critical content, the fact remains that computer 
vision provided art historians with tools that can be used in locating visual 
materials with certain aspects successfully. For instance, Crowley and Zisser- 
man’s retrieval system allowed one to search for simple concepts (e.g. ‘train, 
‘dog’, ‘flower’ ‘bridge’) in painting databases, without requiring annotations for 
these concepts. It works by collecting keyword-indexed images from the inter- 
net and learning from them the appearance of the concept on the fly (2014). It 
became possible to retrieve and visualise paintings of a particular period that 
show a certain visual quality, or contain a certain object or feature. With the 
introduction of style transfer algorithms (Gatys, Ecker and Bethge 2016; Sana- 
koyeu et al. 2018), one more step was taken: the content of a picture could be 
separated from the style of the painting. 

All these steps paved the way to AI algorithms contributing more and more 
to todays’ aesthetic and artistic production and appreciation. We use various 
applications for getting recommendations to the artworks we like, for automat- 
ically ‘beautifying photographs we take, or for assessing aesthetically pleasing 
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photographs with an explanation of the reason behind the assessment, or for 
designing our PowerPoint slides and even for automatic creation of short vid- 
eos from our photographs or videos. The way from the early computer artworks 
to today was a long and winding one. We have seen that within computer art, 
aesthetics was an important research venue. The combinatorial possibilities 
offered by the computer lead the artists to create variations of simple geometric 
patterns, and many possible combinations of a single composition, from which 
the most aesthetically pleasing ones could be selected. With this approach, 
philosophers like Max Bense (Bense 1965) and Abraham Moles (Moles 1966) 
pioneered the search for mathematical rules governing aesthetics, and their 
theories were influential. Today, the aesthetical assessment of images are done 
thanks to the help of image archives that are used in the supervised training of 
machine learning algorithms; in its essence this means that the work ofthe pro- 
grammer has changed tremendously. Instead of ‘programming/coding’ rules 
about aesthetics, the current algorithms are programmed to discover statisti- 
cal patterns in huge image datasets, where the algorithms ‘learn’ by comparing 
images to each other and guessing the correct answer to a question on aes- 
thetics (e.g. which image is more liked by people) or content (e.g. if the image 
has a bird in it) by minimising an error function. The programmer does not 
supervise the learning progress, instead, she provides the algorithm with infor- 
mation about the image data sets. The success of these statistical algorithms is 
simple to assess, they are all widely used. 

If we compare distinct artworks from the earlier era of computer art to Al-art 
of today we might capture the transformation more clearly. Harold Cohen is 
one of the most widely recognised electronic artists, and he let AARON evolve 
through more than 25 years (from 1973 to early 2000s) to its present state of 
maturity. In his words, AARON was originally ‘a program designed to inves- 
tigate the cognitive principles underlying visual representation (Cohen 1988, 
846). In 25 years of its artificial life, AARON ‘learned’ to draw, like a child’s first 
scribbles slowly transforming into a modernist painter’s stylistic abstractions. 
The processes developed by Cohen for AARON to create its paintings can be 
inspected to discover patterns and clues about ‘creativity, but not everyone who 
watches AARON paint will find sufficient evidence to call it ‘creative, nor did 
Cohen ever claim that AARON is creative. There have been debates about the 
definition of creativity, and whether it is possible to concede that an artificial 
intelligence (AI) program can be creative like a painter, or not. After all, if there 
are rules or a procedural description for the artistic activity, then there is no 
reason why a computer program cannot be written to produce art. An impor- 
tant issue here is that many humans contribute to the production of a final 
artwork, and the AI algorithm is not an encapsulated unit, yet the language 
used in their description (e.g. thought vectors, consciousness priors, atten- 
tion) anthropomorphises the algorithms and creates a conceptual problem 
(Epstein et al. 2020). 
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Cohen’s AARON is an early example of AI-based artwork creation, I will visit 
two more recent examples, representing two ends of a spectrum (low-budget 
to high-budget). Bager Akbay’s recent Al-artworks of Deniz Yilmaz, The Robot 
Poet,° took a different approach to the problem at hand, and bypassed the 
definitions of creativity, as well as questions on whether autonomous com- 
putational creativity is possible or not. What Akbay proposed is to generate 
a ‘learning’ poet, just like AARON the painter. However, unlike AARON, the 
underlying program of Deniz Yilmaz is based on the processing of a big data 
set of published poems in a literary magazine, Posta Gazetesi. Akbay expected 
the outcomes of this algorithm to be similar in nature to the dataset, generat- 
ing ‘average’ poems. Similarly, while crafting Deniz Yilmaz’s identity (which 
has its own Facebook page, and now in search for its citizenship), he used 
the photographs that were published along the poems in Posta Gazetesi, and 
generated an average photograph for Deniz. Today, Deniz Yilmaz has mul- 
tiple exhibitions (just like AARON) and a book publication. Like AARON, 
for which Cohen had designed various printing and painting devices, Akbay 
designed a handwriting style and ways of ‘writing’ poems for Deniz Yilmaz. 
But the similarity between AARON and Deniz Yilmaz ends here. Whereas 
Cohen's ambition was to find ways to write a code that learns aesthetic prin- 
ciples, and a way to develop itself, Akbay’s focus was exploring robot rights 
and leading conceptual discussions around the entity of Deniz Yilmaz, asking: 
‘Can a robot poet be considered autonomous, and if so, what are the mecha- 
nisms to enable this transition?’ To that effect, Akbay wrote another algorithm, 
a manager for Deniz Yilmaz’s artistic endeavours, which invited various peo- 
ple to assemble a board of directors to manage Deniz Yilmaz’s dealings within 
the art world. Akbay’s ambition is for Deniz to have a life of its own, where its 
earnings will be transferred to a bank account bearing its own name. He refers 
to Deniz Yilmaz as a failed experiment, as his name as the creator of the robot 
poet still is on the foreground. 

Refik Anadol, who uses big data, as well as big displays to showcase Al- 
artworks, is a well-known name for his (and his team’s) unique approach to 
different data sources and the way he transforms these into new imaginations. 
Anadol explains that he views ‘machine intelligence not only as a new medium, 
but as a collaborator, allowing us to re-examine not merely our external realities, 
but rather an alternative process to which we attribute artistic consciousness’ 
(Anadol 2019). For example, for Latent History,’ a recent work on the history 
of Stockholm, he used archival data consisting of the city’s photographs from 
the last 150 years combined with current photographs. He maintains that the 
classical approaches to displaying such a plethora of data fails short, whereas 
machine learning generates ‘a time and space exploration into Stockholm’s past 
and ultimately present ... on a multidimensional scale’ (ibid.). These kinds of 
explorations make the audience enter a new type of reality along with a new 
type of aesthetics. Still, with every new artwork, even though Anadol claims 
to use machine learning as a collaborator, the artists decisions on what type of 
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data to use for which purposes, and how, renders the results as artworks, not 
the other way around. 

Whereas Cohens, Anadol’s and Akbay’s artworks carry the stigmata of their 
creators, Deep Dream animations generated by style transfer technology have a 
different position. First of all, like the first computer artists, the creators of Deep 
Dream were scientists, and initially, they were not after designing AI algorithms 
to create artworks. As a similar story unfolds, they found the results of their 
algorithms were beyond their expectations, and worthy of investigation: ‘In the 
summer of 2015, we also began to see some surprising experiments hinting at 
the creative and artistic possibilities latent in these models’ (Agiiera y Arcas 
2017). As the resulting ‘artworks’ (see Mordvintsev, Olah and Tyka 2020 for 
examples) are quite different then what is ever hailed as art, the audience was 
both fascinated, and sceptical. From computer art to Al-arts we are still within 
the realm of weak AI, i.e., AI algorithms are used as tools to create artworks, 
and we still see the oscillation between ‘un-sceptical believers and unbelieving 
sceptics, and the impact of the Deep Dream comes from the unexpected results 
it generated, which bestows an autonomous position to the algorithm. 


By Way of Concluding 


Science fiction takes the idea of computers with human capabilities to its 
extremes. However, Jameson points out that science fiction genre is akin to 
utopias, which actually never attempt ‘to represent or imagine a real future 
but rather to denounce our inability to conceive one, the poverty of our imag- 
inations, the structural impossibility of our being able to generate a concrete 
vision ofa reality that is radically different from our current society’ (Jameson 
1982). Lacking the means to open new ways to future realities, science fic- 
tion rather takes on the role to unveil ‘a particular historical present’ (Thacker 
2001, 156). 

When we look at contemporary science fiction productions, especially the 
revisits to old TV series or films such as ‘Battlestar Galactica’ or “Westworld, 
we see that the fear of the strong AI that looks and acts just like humans - but 
stronger and smarter in nature - remains unchanged. When it comes to intel- 
ligent machines, the particular historical presents do not change, even though 
the technology has developed considerably in the past decades, and we live 
in a world where weak AI has permeated into the daily life. As a marketing 
policy, tools and algorithms we use are not necessarily tagged as AI, and this 
might have helped their dissemination without any resistance by the public 
(Tascarella 2020). 

For the interests of what I have presented in this chapter, i.e. the assessment 
of artworks generated with/by computers and/or computational creativity, 
we see the same trend: the audiences, as well as the creators still prefer to be 
amazed by the unexpected results of human-machine collaborations (like in 
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Deep Dream), and are not so much interested in bestowing an autonomy and 
creative status on the programs themselves (like in the case of Deniz Yilmaz), 
but they are much more open to new imaginations that are created with AI 
(like in the case of Latent History). Throughout the chapter, I have referred 
to computer art and Al-arts as two separate art movements. However, when 
we follow the broad definition of computer art, ie. artworks generated by art- 
ists/scientists with the aim of challenging the boundaries of arts as well as sci- 
ences, we see that Al-arts still fall under the umbrella of the latter. Of course, 
a lot has changed since the first computer artworks, in intention, goals and 
challenges. More importantly, today, Al-arts offer a platform where computers 
could become more than tools, and collaborators, and maybe in the future, sole 
artists. As Mark Weiser (1995) in his now famous Scientific American article 
noted, ‘the most profound technologies are those that disappear. They weave 
themselves into the fabric of everyday life until they are indistinguishable from 
it, and the weak Al creativity already disappeared. The question remains when 
the strong computational creativity will achieve the same status, and how it will 
be hailed when it does. 


Notes 


1! In 2003 only, Menzies (2003) listed topics where Al-research was suc- 

cessfully implemented when he described the road ahead. He empha- 

sised the ubiquity of the tools AI researchers could use and combine. 

Today, the tools mentioned in that list are already operational in daily life. 

These tools are considered successful as weak AI examples: they operate 

fairly well on the tasks they are designed to accomplish, but they do lack a 

general intelligence. They cannot do anything but what they are designed 

to do. 

For a historical overview of AI, please see Russell and Norvig (2002). 

For a detailed history of computer art see Franke (1971) and Noll (1994). 

This section summarises research on technoscience art (Akdag Salah 

2008). 

To see various artworks generated by AARON, please visit: http://www.aar 

onshome.com/aaron/index.html. 

To see the Mondrian and Noll’s artworks used in the experiment, please 

visit: http://dada.compart-bremen.de/item/artwork/5. 

To see the list of exhibitions of the robot poet Deniz Yilmaz, please visit: 

https://www.poetryinternational.org/pi/poet/29478/Deniz-Yilmaz/en/tile. 

You can access Yilmaz’s published book from here as well: https://drive 

.google.com/drive/folders/0B6I_wTbmgoBMeUp20ExhQVVUWEU. The 

poems are generated in Turkish. 

7 To see a sample of Latent History, please visit: https://www.fotografiska 
.com/sto/en/news/refik-anadol-latent-history 
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AI Power and Inequalities 


CHAPTER | | 


Primed Prediction: A Critical 
Examination of the Consequences 
of Exclusion of the Ontological 
Now in AI Protocol 


Carrie O’Connell and Chad Van de Wiele 


The dominance of the machine presupposes a society in the last stages of increasing 
entropy, where probability is negligible and where the statistical differences among 
individuals are nil. Fortunately we have not yet reached such a state. 


- Norbert Wiener (1989, 181) 


Introduction 


Norbert Wiener (1989) concludes his seminal work, The Human Use of Human 
Beings: Cybernetics and Society, with a warning. The thermodynamic universe, 
as he envisioned it, was evolving towards an entropic fate, as natural systems 
do. As entropy and progress are at odds, and ever the champion of purposive 
progress, Wiener applies the Darwinian principle of natural selection as a guide 
for a progressive cybernetic future. Wiener’s concept of negentropy, or the miti- 
gation of such natural entropic determination (Faucher 2013), is premised on 
the optimism that tailored feedback within cybernetic systems could teach 
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machines to redirect course towards more organised and error-reduced (rather 
than error-free) outcomes. The point of such tailoring - in the sense that it 
serves as a blueprint for algorithmic prediction - is to model possibilities of 
human behaviour relating to the socio-cultural. However, at what point does 
the simulation of human behaviour become just a more consumable way of 
saying, ‘shaping behaviour through technology’? 

The primary purpose of this chapter is to explore the shortcomings of modern- 
day applications of Wiener’s cybernetic prediction - the theoretical foundation 
of artificial intelligence (AI) - particularly in terms of capture technologies 
that remain ubiquitous as a method of data collection for feeding such sys- 
tems. We argue that such data are not impartial or necessarily explanatory, 
but rather evidence of third-order simulacra, simulation, as conceptualised by 
Jean Baudrillard (1994). We examine what cybernetic prediction, as outlined 
by Wiener, excludes; namely, an attendance to the complex ontological now, 
which Baudrillard warned against in his analysis of the order of simulacra - 
particularly the role technological innovations play in untethering reality from 
the material plane, leading to a crisis of simulation of experience. Secondly, 
we explore the potential psychosocial consequences associated with machine 
learning systems predicated on a cybernetic theorem that foundationally relies 
on human repetition - specifically, that reliance upon such repetition leads to 
the very entropy that Wiener warned against. As Mumford (1972) notes in his 
essay, Technics and the Nature of Man, human nature may be subsumed, ‘if not 
suppressed’ (77), by the technological organization of intelligence into techno- 
logical systems. From this perspective, any machine learning system rooted in 
Wiener’s view of cybernetic feedback loops risks creating outcomes through a 
process of subjective priming, more so than predicting it. 


The Genesis of Cybernetics 


Emerging mid-century, and inspired - in part - by the technological advance- 
ment ofboth machinery and intelligence-gathering systems that emerged during 
WWII, Wiener’s theory of cybernetics focuses on the diffusion of communica- 
tion in terms of control imposed by constraints and allowances afforded by 
the networks through which messages spread. Inspired by the 17th-century 
philosopher and progenitor of modal metaphysics, Gottfried Leibniz - in part 
because of Leibniz’s explication of language as a computational system, and in 
part due to Leibniz’s fascination with the potential of automata - Wiener envi- 
sioned a system of feedback in which man and machine are indistinguishable 
when considering message input and output. Like living organisms which have 
‘a tendency to follow the patterns of their ancestors’ (Wiener 1989, 27), cyber- 
netic systems, too, in their ability to be shaped by external stimuli, can leverage 
feedback as a ‘method of controlling a system by reinserting into it the results 
of its past performance (61). The past, in other words, can inform and correct 
future outcomes. 
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To Wiener, the goal of understanding communication feedback as a compu- 
tational system wasn't simply to reflect upon the human condition, but leverage 
that reflection as a tool of prediction for future events. The ‘divine intermediary’ 
in Wiener’s calculation wasn't a Leibnizian pre-established harmony ordained 
by God or natural law - these were the prescriptions that lead to the entropy he 
warned against. Instead, the intermediary would take human form, ordained 
by a prescription of diverse input from not just the scientist, but also the ‘phi- 
losopher and anthropologist. From a 21st-century perspective, with decades of 
applied cybernetic prediction as evidence, it is necessary to wonder, however, if 
the very entropy Wiener warned against has come to fruition via the exclusion 
of input variety by those who design, operationalise, and ultimately capitalise 
on predictive technologies which surveil, capture, and predict human behav- 
iour and events. 

One focus of contemporary concern regarding the application of cybernet- 
ics is that, as the theoretical foundation for AI, its principles are often applied 
beyond the ‘negligibly small’ domain of truly closed systems. As Faucher (2013) 
argues, “The utility of cybernetics is confined to very local and specific contexts, 
and in a universe of increasing complexity, cybernetics will not necessarily save 
us (206). Yet, today, cybernetic principles undergird algorithms designed to 
predict everything from global economies to recidivism in the arena of crimi- 
nal justice. The question, however, is whether such cybernetic-based systems 
objectively reflect potential probability in an effort to prune towards pro- 
gress, or ‘play an active role in steering the likelihood of an event’ (Faucher 
2013, 211), thereby priming behaviour, both machine and human, towards 
future outcomes. 


The Algorithm: Third-Order Simulacra 


Wiener analogizes machine learning to the neurological process of receiving 
input, stimulating synaptic flares, recording memory (or, taping), and ulti- 
mately evolving future responses to stimuli. To explain this taping mechanism 
‘which determines the sequence of operations to be performed, (65) he refers 
to the recreation of this physiological function in digital form as the ‘mechani- 
cal simulacra of the brain, (65). This function of the human brain provides an 
apt blueprint for Wiener’s vision of machine learning on two fronts. On the one 
hand, the analogy provides an elegant heuristic for understanding the learning 
process in easily accessible terms. On the other, it reminds us that the neuro- 
logical process of recording memory is hidden from plain sight - shrouded 
by a vessel of skin and bones, nerves and blood flow. In mechanical terms, 
this shrouding, or ‘black-boxing, is done via bits and code. Cautious of the all- 
or-nothing binary that might be gleaned from this analogy for learning, Wiener 
recognised that we must treat the human subject as a cultural creation, not 
just an agent of neurotransmission that records memory, or data, to be ana- 
lysed. However, as machine learning has advanced, the genesis of such cultural 
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creation is called into question. As Martin Hand (2014) notes, Algorithmically 
produced data now accesses us, intervening and mediating nearly all aspects of 
everyday life whether we know it (like it) or not’ (8). Thus, a new social ontol- 
ogy has emerged, consistent with Baudrillard’s definition of third-order simu- 
lacra: We exist in a ‘dataverse’ in which the world is literally made of data - so 
too, our cultural knowledge. 

As Baudrillard (1994) describes, there are three orders of simulacra - which 
can be historically paralleled against the epochal transitions from the pre- 
Industrial, to Industrial, to Digital eras, and manifold scientific advances which 
anchored each. The first order comprises those simulacra which are natural- 
ist, counterfeit images of reality that still aim for the restitution or the ideal 
institution of nature made in God’s image’ (Baudrillard 1994, 121). In the pre- 
Industrial Age, the nature of being, as inspired by God, defined natural reality 
and universal truth. The second order of simulacra are materialised as prod- 
ucts, made possible by the advanced machinations of the Industrial Age, which 
generated the expansion of globalisation. Scientific invention, as materialised 
by the machine, prominently figured as a technical form of magic in the scien- 
tific imagination during the Industrial era. The imbued power of God which 
had defined ontological reality in the pre-Industrial mind was now replaced 
with the power of science fiction premised on a future made possible by the 
Promethean power of industrial technologies. The third order, and arguably 
the most confounding, are the simulacra of simulation - that which is ‘founded 
on information, the model, the cybernetic game’ (121), and whose aim is total 
operational control. 

Fundamentally, Baudrillard’s explication of the order of simulacra is a quest 
for the provenance of ontological truth. Due to the emergence of the techno- 
logical ‘other’ in the form of simulation, we are on the precipice of a cultural 
hiatus, distortion, or rift of ontology. Today, ‘truth’ has been subsumed into 
a self-referential system of binary code by those who seek to operationalise, 
predict, and ultimately control human behaviour. Such cybernetic ‘truth’ is 
not inspired by nature (vis-a-vis ‘God’), or the Modern principles of human 
imagination that provoked scientific inquiry, but feedback loops that selectively 
include and exclude data input for reasons obscured or ‘black-boxed’ from the 
end-user. The power of God that defined ontological reality in the first order of 
simulacra, as well as the power of scientific imagination that defined ontology 
in the second, has now been firmly replaced by a new mode of instantiation - 
the algorithm. 

An investigation of Baudrillard’s concept of simulation to explore the power 
imbalance created by modern technology is not without precedent. In her 
book, Paper Knowledge: Towards a Media History of Documents, Lisa Gitelman 
(2014) examines the troubled ethos behind digital simulation - the site of 
the disappearance of meaning and tangible representation. Similarly, Castillo 
and Egginton (2017) argue that, in the digital era, what is ‘rea? and what is 
a constructed ‘copy’ has become increasingly difficult for the human user to 
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distinguish due to the black-boxed, bits-based nature of production in a cyber- 
netic world. Similarly, in his analysis of the legacy of the Automaton Turk on 
current perceptions of AI, Ashford (2017) notes that machines are capable 
of ‘projecting illusions that can undermine our very ontologies’ (139), and 
suggests that computational technology might soon eclipse human agency 
in shaping history. Uricchio (2017) echoes this concern in his analysis that 
what defines subject (human) and object (technological artefact) has been 
confounded by modern-day algorithmic intermediaries that are capable of 
self-learning. In other words, in the 21st century, as machine learning evolves, 
authorship of - not just output, but the system itself - has been taken from the 
hands of humans who have become passive contributors of data. Soon, algo- 
rithms will know so much about our behaviour that such agency will no longer 
be foundational to the cybernetic relationship between a technical system and 
human interlocutor. 

In many respects, Wiener envisioned this algorithmic future. Fascinated by 
the idea that black boxes, or those cybernetic units ‘designed to perform a func- 
tion before one knew how it functioned’ (Galison 1994, 246), Wiener - in the 
philosophical vein of Descartes - thought it possible to create hardware that 
replicated the function of the human brain. As Jeffrey Sconce points out in The 
Technical Delusion, Wiener himself envisioned a ‘brain-in-a-jar’ form of cyber- 
netics: “Theoretically, if we could build a machine whose mechanical structures 
duplicated human physiology, then we could have a machine whose intellectual 
capacities would duplicate those of human beings’ (Wiener as cited in Sconce 
2019, 234). Yet, as Galison (1994) notes, critics of Wiener’s black box project 
saw the potential for ‘the elimination of inner states of human intention, desire, 
pleasure, and pain in favour of purely observable manifestations’ (252). At the 
heart of cybernetic prediction is the belief that to understand human beings, it 
is first essential to understand how patterns of information are created, stored, 
retrieved, and organised (Hayles 2008). However, such cybernetic prediction is 
a narrow, self-referential system focused on the past and future in which infor- 
mation input plays a privileged role in hiding ‘the real behind a veil of digital 
representations designed to take command of life itself? (Faucher 2013, 211). 
And, as Hand explains, “The dataverse promises a new descriptive-predictive 
analytics of pattern and correlation, prioritized over meaning and causation 
(2014, 10). That is, rather than producing meaning, algorithms - black boxes 
that house and take as input information that becomes simulatory - merely 
produce more information. 

Critically, this process and the technical systems that facilitate it closely align 
with what Philip Agre (1994) describes as capture. According to Agre, capture 
serves as both a linguistic metaphor (opposite the visual metaphors of surveil- 
lance, as articulated by Orwell and Foucault) and material process of tracking 
used to characterise the institutional, technical logic whereby human activities 
are captured and represented, or tracked, within sociotechnical systems. Cap- 
ture technologies, Agre explains, comprise five interlocking processes through 
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which sociotechnical systems represent, constitute, direct and/or transform 
human activity through its purported ‘discovery’; these processes include: 
(1) analysis, (2) articulation, (3) imposition, (4) instrumentation, and (5) elabo- 
ration. First, the activity in question is analysed and ontologically rendered into 
basic, programmatic terms (objects, relations, variables, etc.) for the subsequent 
articulation of grammars of action, which delineate ‘the ways in which those 
units can be strung together to form actual sensible stretches of activity’ (Agre 
1994, 746). Next, these grammars are socially and/or technically imposed upon 
those engaged in that activity (i.e. made legible by the capture system) and 
recorded via some means of instrumentation. Lastly, captured records of that 
activity may be elaborated upon (audited, modelled, merged, stored) for opti- 
misation. Capture, as Agre clarifies, may thus be deployed for either the archiv- 
ing of data as input and/or the abstraction of ‘semantic notions or distinctions, 
without reference to the actual taking in of data (744), as with AI-based 
systems. Thus, as Chun (2016) explains, ‘An AI program has successfully 
“captured” a behaviour when it can mimic an action ... without having to 
sample the actual movement’ (59-60). 

As Malik (2010) argues, however, ‘control in the cybernetic sense does not 
mean absolute control of every detail. It is more like steering, directing and 
guiding’ (33). To aid in this guidance requires a broad brush applied to cull 
information into categories. Take, for instance, AI-based risk assessments - 
built upon the fallible premises of cybernetic prediction - that are accu- 
rate only insofar as they produce risk as simulation via capture (i.e., of past 
behaviour) by categorising individual risk in terms of broad sociological data. 
Cathy O’Neil (2016) describes various public and private domains within 
which predictive models obscure - and ultimately magnify - human bias, 
such as the use of recidivism software for criminal sentencing just men- 
tioned. As O’Neil argues, ‘sentencing models that profile a person by his or 
her circumstances, including socioeconomic status and familial/social ties, 
‘help to create the environment that justifies their assumptions’ (O’Neil 2016, 
29). Accordingly, the risk of recidivism is primed using narrow parameters 
that often exacerbate racial and class-based disparities. In a recent interview, 
Wendy Hui Kyong Chun similarly discusses the proclivity for credit monitor- 
ing systems to reify the purported ‘risks’ they aim to detect and avoid (i.e., 
[in]ability for repayment; Chun and Cotte 2020). Based on various factors 
(beyond the borrower’s credit/financial history, such as educational attain- 
ment and social network ties, etc.) risk assessment models designed to predict 
creditworthiness are, in effect, programming the very conditions they claim 
to eschew - an outcome of benevolent surveillance described elsewhere by 
Marion Fourcade and Kieran Healy (2007; 2017). What these cases demon- 
strate is the relationship between capture and risk, whereby risk as simulation 
becomes embedded within technical systems of capture intended to predict 
and mitigate future risk. 
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The Self-Referential Learning Machine 


From earlier approaches to AI (i.e., ‘expert systems’ built upon ‘if-ther rules with 
limited scalability), presently dominant approaches rely upon unsupervised 
machine/deep learning, leveraging information theory and connectionism 
for scalable prediction and decision-making (for a comprehensive discussion 
of AI paradigms and their evolution, see Russell and Norvig 2016). Among 
the myriad public and private domains wherein these Al-based systems prime 
social outcomes, perhaps the most consequential and ethically questionable 
is the criminal-legal system. In the U.S., algorithmic decision-making pro- 
grams, predictive policing applications, and targeted/anticipatory surveillance 
technologies have become standard fare. Wiener recognised the potential for 
human actors - governments, militaries, and other cultural hegemons - to lev- 
erage the power of the learning machine against its citizenry, and cautioned as 
much. To mitigate such domination - both of the machine and the human 
actors who seek to leverage its power, Wiener (1989) heeds that ‘we must know 
as scientists what man’s nature is and what his built-in purposes are, even when 
we must wield this knowledge as soldiers and as statesmen; and we must know 
why we wish to control him’ (182). It is not just the scientist, he notes, that 
should be responsible for our new technological future, but also the anthro- 
pologist and philosopher, if we are to prevent such an entropic reality. 

Complicating the relationship between information input and predictive 
outcomes is the problem of data categorization that is foundational to capture 
technologies. For example, as applied to risk assessments for criminal offend- 
ers, a qualitative understanding of the perpetrator, as well as those individually 
particular antecedents which may have factored into the commission of a par- 
ticular crime, are secondary (if considered at all) to the broad categories within 
which a perpetrator may fall. Data such as age, race, and socioeconomic status 
are far more valuable to the cybernetic game because they may be reduced to 
easily quantifiable statistics. The propensity for AI-based, cybernetic systems 
to prime (i.e., prune) human behaviour has been explored by several schol- 
ars, albeit in different ways: From reproducing essentialist social categories 
and magnifying their attendant (institutional, economic, etc.) disparities, to 
transposing notions of risk and the institutional handlings thereof. In Coming 
to Terms with Chance, for instance, Oscar Gandy Jr. (2009) describes cross- 
sector technologies of ‘rational discrimination’ that ‘facilitate the identifica- 
tion, classification and comparative assessment of analytically generated 
groups in terms of their expected value or risk’ (55). Such techniques, leverag- 
ing actuarial risk models and statistical evidence for purposes of prediction, 
serve to emphasise and reify race as an essential category (via proxy measures; 
see also Harcourt, 2015). 

Cybernetics, at its core, is the acute science of subjective choice reduction 
as a means of avoiding entropy, which makes such categorization attractive. 
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As Faucher argues, ‘Cybernetics does not drive toward the ultimate truth or 
solution, but is geared toward narrowing the field of approximations for bet- 
ter technical results by minimizing on entropy’ (2013, 206). Yet, as modern 
applications of algorithmic and AI-based risk assessment systems illustrate, 
the push towards determining a predicted ‘truth or ‘solution’ has achieved the 
opposite, partly due to the reliance upon categories of data - rather than a 
variety - as the heuristic which guides machine learning. Wiener (1989) 
illustrates the value of variety of external input in digital systems, warning 
that closed systems run the risk of homogeneity, thereby increasing entropy, 
or a devaluation of output. To illustrate this point, and simultaneously argue 
that systems will only be as good as their human creators make them, Wiener 
envisions a digital remaking of Maelzel’s chess-playing Automaton Turk as an 
example of where the future of machine learning may lead, if variety in exter- 
nal output is considered: 


A chess-playing machine which learns might show a great range of per- 
formance, dependent on the quality of the players against whom it had 
been pitted. The best way to make a master machine would probably be 
to pit it against a wide variety of good chess players. (177) 


His reference to the Automaton Turk is quite apt, as it is seen both in its day and 
in hindsight as an iconic example of technological deception at the hands of a 
skilled human operator, able to fool the audience based on both sleight of hand 
theatrics, as well as a keen insight into predictable human behaviour. 

To Wiener (1989), exposing a novel computerised version of the ‘Turk to a 
variety of chess master challengers offers hope that the system can learn from 
mistakes, recalling past defeats in an effort to not repeat them. This exposure 
to variety, thus, unburdens the chess-playing automaton - once the controlled 
object of a single human operator - from its storied narrative of being nothing 
more than an inauthentic representation of communicative exchange between 
subject (human audience) and machine object. The machine may escape an 
entropic fate by gaining new information via the continued interaction with a 
variety of experts. Yet, from a 21st-century perspective, Wiener’s optimism falls 
short two-fold: (1) machine learning is capable of self-propagation (Uricchio 
2017), reducing the role of human input to that of passive data source, rather 
than active participant in the creation of knowledge, and (2) the basis of 
machine learning as Wiener envisioned it - that of cybernetic feedback loops 
informed by past action to predict future outcomes - allows for applied inter- 
pretations that dismiss present context (Halpern 2014). Additionally, the pro- 
duced information output itself - conforming to a grammar of action imposed 
to maintain ‘compliance between system records and ongoing events’ (Agre 
1994, 748) - is reified as truth, rather than simply more information. It is this 
reification, evidenced in the practice of risk assessment technologies, that steers 
the use of these technologies away from the aim of cybernetic negentropy and 
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towards what Wiener cautioned against: homogeneity within the closed system 
that will ultimately undermine it. 


Capturing Behaviour, Programming Risk 


The computers won, but not because we were able to build abstract models 
and complex situations of human reasoning. They bypassed the problem of 
the agent's inner life altogether. The new machines do not need to be able 
to think; they just need to be able to learn. 

- Fourcade and Healy (2017, 24) 


In order for machines to learn, they must be able to correct prior errors. In 
order to correct such errors, those missteps must be recorded as feedback 
in order to inform the feed-forward. The philosophical underpinning as to why 
and how such errors can be recorded stems from Wiener’s assessment of bio- 
logical memory as the by-product of synaptic flares that imprint on the human 
mind due to the physiological gravity of experience. In other words, memories 
stick - and may even aid in shaping how we approach future events - when 
they are derived from heightened sensory experience. As an analogy: I may not 
recall what I ate for breakfast on an otherwise insignificant and random day a 
decade ago, but I can tell you precisely the colour of the bike and the sensation 
of pain that I experienced when first riding and crashing a bicycle. It is not the 
narrative of the event that imprints the memory, but the connection of that 
event to a physiological sensation experienced emotionally or tactically. It is 
this reflection upon past experience that paves the way for understanding the 
mind monad as a system of learning, which Wiener believes could be replicated 
in machine form. 

Like Leibniz, Weiner qualifies the relationship between the mind and experi- 
ence (past and present) as a communicative process, though goes a step fur- 
ther to suggest that ‘the organism is not like the clockwork monad of Leibnitz 
with its pre-established harmony with the universe, but actually seeks a new 
equilibrium with the universe and its future contingencies’ (Wiener 1989, 48). 
Simply put, like the pruning of Darwinian natural selection, the potential for 
robust cybernetic systems to weed-out frailties in the organism prepares, or as 
we argue, primes the subject for future environments. 

Unfortunately, the data upon which these systems operate are often biased, 
incomplete or simply unqualified. For example, in the sentencing of convicted 
criminals, factors beyond the individual’s crime - such as broader recidivism 
rates based on socioeconomic and demographic data - are used to predict the 
likelihood an individual may be a repeat offender, thereby influencing sentenc- 
ing (Hillman 2019). Accordingly, it is fair to question whether such potential 
‘predicted’ outcomes are primed via the algorithmic encoding of emotional 
triggers Weiner believed encouraged behavioural repetition. 
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As Halpern (2014) argues, the basis for Wiener’s belief in the possibility of 
prediction is that humans, under duress, act repetitively. When applied to law 
enforcement, this logic produces an ostensible feedback loop whereby, for 
instance, statistical models based on prior (individual) arrest rates - already 
contaminated by racial/demographic assumptions vis-a-vis crime (e.g., over- 
policing of Black neighbourhoods; see for example Crawford 2018; Pasquale 
2015) - ‘generates the data that validate its hypotheses about race without 
necessarily involving animus based on features unrelated to criminal behav- 
iour (Gandy 2009, 125). In such an algorithmic scheme - aptly described by 
Frank Pasquale as a ‘reputation system’ - based on cybernetic principles of 
prediction, the individual is reduced to mere data points of past behaviour 
coupled with macro-level sociological data in a decision-making feedback 
loop bereft of present context. In the language of capture, this produces a 
grammar of action that reorients and superintends - through imposition 
and instrumentation - the activities of those within a given socio-technical 
system (in the case of law enforcement, both officer and suspect); that is, 
human activity becomes systematised around a standard ontology for ‘main- 
taining the correspondence between the representation and the reality’ (Agre 
1994, 742). 

To elucidate the psycho-social consequences of this process, we consider 
data policing/management programs and actuarial risk assessment tools for 
criminal sentencing, as these most readily clarify the notion of risk as simula- 
tion; indeed, as Harcourt (2015) explains, ‘risk today has collapsed into prior 
criminal history’ (237). Examples of these tools - particularly in the United 
States - are innumerable and continue to gain traction among state and fed- 
eral law enforcement agencies. Introduced by the New York Police Department 
(NYPD) in 1995, CompStat (short for computer and/or comparative statistics) 
was developed to capture and index, in real-time, crime-related data that law 
enforcement may use to inform and direct policing efforts (Bureau of Justice 
Assistance 2013). Similarly, PredPol' was developed by the Los Angeles Police 
Department (LAPD) and researchers at the University of Southern Califor- 
nia in 2012 (PredPol n.d.). Unlike CompStat, which initially relied only upon 
historical (macro-level) crime data to track and prevent crimes, PredPol was 
designed to anticipate when and where crime might occur (PredPol n.d.). Since 
their inception, the prevalence, sophistication and purported accuracy of these 
and similar tools has increased: As of 2016, 20 of the 50 largest law enforce- 
ment agencies in the US reported using at least one form of predictive policing 
(Jouvenal 2016), demonstrating a broader shift toward ‘algorithmic govern- 
ance’ and data policing (Završnik 2019, 2). That is, a reliance upon automated 
data analysis and prediction (e.g., via Al-based systems) for decision-making 
by law enforcement and intelligence agencies, which, according to Završnik, is 
supported by the neoliberal emphasis on objectivity, legitimacy and efficiency 
(see also Benbouzid 2016; Wang 2018). 
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Presented as an affordable and reliable solution to limited police resources, 
predictive policing, by all accounts, appears neutral and accurate. In 2013, 
following the forced downsizing of the police department in Reading, 
Pennsylvania, police chief William Heim implemented PredPol in order to 
streamline law enforcement efforts; one year later, the number of reported bur- 
glaries decreased by 23 percent (O’Neil 2016). Despite this and other seem- 
ingly positive outcomes, critics have warned of the potential consequences 
of predictive policing programs; namely, for their reliance upon skewed and 
self-reinforcing crime-related statistics from the over-policing of communities 
of colour (Ferguson 2017; Hinton 2016; Jouvenal 2016). That CompStat has 
been critically associated with the ‘broken windows theory of policing’ further 
clarifies the inherent social biases and prejudicial animus - whether implicit or 
overt - embedded within such tools and their attendant practices (e.g., Eterno 
and Silverman 2006). 

Consider the following scenario: In an effort to stymie crime in a poverty- 
stricken, urban neighbourhood - itself a historical product of multi-layered 
and intersecting patterns of social and economic disenfranchisement, usually 
along racial lines - police engage in round-the-clock patrols of that area. As a 
result, and by virtue of institutionalised pressure to tangibly reduce crime 
(Giacalone and Vitale 2017), police stops and summonses become more fre- 
quent. Consequently, reported crime rates for that neighbourhood increase, 
feeding police management databases and prediction tools context-deprived 
data points, thereby prompting further patrols, arrests and so on, thereby trig- 
gering ‘cascading disadvantages’ (Pasquale 2015). Thus, the ‘CompStat men- 
tality’ (Giacalone and Vitale 2017) - impelled by blind faith in the capture/ 
analysis of quantitative data, corresponding to the neoliberal underpinnings of 
‘algorithmic governance’ (Završnik 2019) - may be understood as a grammar 
of action that primes law enforcement toward decontextualised metrics of pro- 
ductivity, obscuring the connotations of physical violence within the ‘capture’ 
metaphor (Agre 1994). As crime becomes untethered from its social dynam- 
ics through this grammar of action, so too does law enforcement become 
estranged from the communities it claims to serve and protect. 

Unlike crime management tools (e.g., CompStat) and predictive policing soft- 
ware (e.g., PredPol), which, at their core, aim to ‘prevent’ criminal activity by 
forecasting who is most likely to commit what type of crime, when and where - 
using historical crime data - criminal risk assessment programs assess the like- 
lihood of recidivism (i.e., that a convicted criminal will re-offend). According 
to Carlson (2017), such tools include ‘actuarial instruments, or models that 
predict risk of recidivism by studying the common traits of paroled inmates 
responsible for committing multiple crimes’ (305). Although many risk assess- 
ment programs available today rely upon AI and algorithmic models, predictive 
assessments of criminal risk have been used in the US since the 1930s* and have 
steadily gained traction among law enforcement since (Harcourt 2007). In fact, 
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the National Institute of Corrections, a subdivision of the US Justice Depart- 
ment, encourages law enforcement agencies to incorporate risk assessments at 
each stage of the legal process (Angwin et al. 2016). Given the seeming poten- 
tial to reduce incarceration rates and correctional costs by ranking offenders 
according to probable threat (Harcourt 2015), risk assessment is among the 
leading forms of predictive decision-making within the criminal justice system. 

Investigations of risk assessment programs and their outcomes, however, 
have revealed the very inequities and ethical issues detailed earlier (e.g., 
Harcourt 2007; 2015; O’Neil 2016). As Casacuberta and Guersenzvaig (2018) 
explain, the utlisation of these algorithms is predicated upon an assumption of 
fairness and objectivity, though such outcomes are not necessarily guaranteed. 
Take, for example, the Correctional Offender Management Profiling for Alter- 
native Sanctions (COMPAS) risk assessment tool - an extension of the Level 
of Service Inventory (LSI), the leading risk assessment instrument among law 
enforcement agencies (Angwin et al. 2016; Northpointe 2015; Wykstra 2018). 
In 2016, an investigation conducted by ProPublica revealed the degree to which 
implicit racial bias impacted risk assessment scores via COMPAS (Angwin et al. 
2016; Wykstra 2018). Using results from over 7,000 arrestees in Broward 
County, Florida, Angwin and colleagues reached several conclusions: Not 
only were risk assessment scores unreliable for projecting violent crimes, they 
were also unevenly distributed between Black and White defendants. As the 
researchers concluded, COMPAS ‘was particularly likely to falsely flag Black 
defendants as future criminals, wrongly labelling them this way at almost twice 
the rate as White defendants’ (Angwin et al. 2016, para. 15). Perhaps unsurpris- 
ingly, Northpointe, the for-profit organisation behind COMPAS, maintains that 
the program does not consider racial categories in calculating risk; however, as 
Harcourt (2015) argues, other factors included within risk assessment mod- 
els serve as proxies for race. Specifically, the COMPAS model, as well as other 
risk models such as LSI and LSI-R (the Level of Service Inventory-Revised), 
includes educational attainment (both of the individual and their family mem- 
bers), employment status and income, and prior criminal history in determin- 
ing risk, which distribute unevenly along racial lines and thus reflect - and 
augment - the pathologising effects common among earlier policing practices 
(see Hinton 2016). O’Neil (2016) aptly illustrates this scenario: 


A person who scores as ‘high risk is likely to be unemployed and to 
come from a neighbourhood where many of his friends and family have 
had run-ins with the law. Thanks in part to the resulting high score on 
the evaluation, he gets a longer sentence, locking him away for more 
years in prison where he’s surrounded by fellow criminals - which raises 
the likelihood that he'll return to prison. He is finally released into the 
same poor neighbourhood, this time with a criminal record, which 
makes it that much harder to find a job. If he commits another crime, 
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the recidivism model can claim another success. But in fact, the model 
itself contributes to a toxic cycle and helps to maintain it. (27) 


As this reabsorption makes clear, risk outputs derived from algorithmic and AI- 
based programs - designed to analyse, predict, and mitigate or prevent future 
damages (harms, losses, etc.) - do little beyond programming and, arguably, 
ensuring future risk; particularly when risk scores are introduced during crimi- 
nal trials. In bypassing crucial facets of the human experience and other exog- 
enous factors (e.g., prison cycling), risk is reduced to a sequence of quantitative 
variables - a grammar of action imposed upon those whose activities have been 
captured (Agre 1994) - that, taken together, merely (re)produce the hyperreal, 
the imaginary, and the immanent. As Baudrillard (1994) suggested in delineat- 
ing third-order simulacra, “The models no longer constitute the imaginary in 
relation to the real, they are themselves an anticipation of the real, and thus 
leave no room for any sort of fictional anticipation’ (122). That is, in attempting 
to model and predict risk (of recidivism), the gap between real and imagined 
risk yawns, producing risk as simulation - an imitation of risk simulated and 
reabsorbed through retrospective cybernetic systems and practices thereof. 


Conclusion: Lifting the Cybernetic Veil 


Wiener acknowledges the potential for the abuse of cybernetic systems by 
external forces when he warns that ‘the machine's danger to society is not from 
the machine itself, but what man makes of it’ (1989, 182). “The great weakness 
of the machine’ (1989, 181), he states - the weakness that would prevent the 
domination of humankind by machines and, subsequently, those human agents 
who seek to leverage the power of cybernetics for control over populations - is 
that the machine itself cannot account for the myriad conditions that qualify 
human existence. Leibniz considered these myriad conditions, as well as future 
possibilities, to be contingencies that were accounted for, organised and even 
predicted by a pre-established harmony vis-a-vis God, or divine intermedi- 
ary. To Leibniz, the power of his new cybernetic ‘calculus’ of communication 
was as a tool of ontological reflection; however, this is where Wiener breaks 
from Leibniz. 

In order to understand our current path towards a socio-psychological 
entropic fate at the hands of cybernetic prediction, it is necessary to reflect 
upon the gravity of the philosophical detour Wiener (1989) takes from the 
‘patron saint of cybernetics. Leibniz, like Descartes before him, made early 
headway into the question of substance dualism, or the distinction between 
the mind (the thinking substance) and body (the extended substance) as 
separate, though dependent, entities. To Descartes, these created substances 
are relational, working in perfect union of mind and matter to form the sub- 
ject. What distinguishes Leibniz’s approach to the mind-body problem is his 
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rejection of the notion of the body as extended substance, and therefore subject. 
In his theory of monads - or, simple, unextended substances - Leibniz agrees 
with Descartes that the mind (or soul) qualifies as substance, or monad. Mon- 
ads are independent from causal extension; therefore the body does not qualify. 
Additionally, a monad’s properties are naturally continually active, changing, 
and evolving over time. 

On the surface, this argument seems contradictory to his rejection of Carte- 
sian dualism. If both the mind and body evolve, how can they not be both seen 
as substance? The answer lies in Leibniz’s definition of the natural world. As de 
Mendonça (2008) states in her explication of Leibniz’s concept of nature, Leibniz 
distinguishes between material nature, or that ‘which is produced in nature 
according to mechanical principles, and that which is natural to the soul, ‘and 
explained by its own principles - namely, the principle of perfection’ (187). The 
distinction between mind and body, then, is found in the genus of each. 
The soul, as natural perfection, is created by God. The body, as material form, is 
merely organised and transformed by the laws of nature. In this sense, the mind 
and body are not equal, causal entities; rather, the mind, or soul, is the ultimate 
conductor of the subject. 

The implications to current applications of cybernetics, in general, and AI, 
specifically, are thus called into question. To see the body as extended substance, 
as Wiener did, provides the philosophical foundation upon which one can jus- 
tify the mechanical object as a replication of the human brain. The cybernetic 
brain-computer model, as Sconce (2019) notes, while perhaps deeply flawed in 
analogy and application, is something ‘we all believe’ (235) to be self-evident. 
Echoing Hayles (2008), Sconce (2019) continues: ‘Underlying the cybernetic 
dream of uploading consciousness is a magical positivism born of a panicked 
materialism, a belief that any and all questions can be resolved through the 
accumulation of sufficient data (235). Yet, as the aforementioned examples of 
cybernetic risk assessment illustrate, this accumulation of data is often far from 
sufficient, and more often than not subjectively reductive. 

Perhaps it is time to revisit the mind-body problem as it relates to cybernetic 
principles, and explore the merits of predictive technology from this philo- 
sophical foundation. In his new calculus, Leibniz introduced the mind-body 
problem that ‘included the new concept of the differential within the field of 
significations’ (Serfati 2008, 127). To Leibniz, meaning is a complex negotiation 
between both what is tangibly present, tangibly missing, and the qualitative sig- 
nificance of that difference. External ‘substances’ therefore, cannot be regarded 
as true subjects, but rather as modes or states of presentations of an assemblage. 
By biasing towards the thesis that the mind (or soul) is the single natural source 
of human substance, and everything else an ever-evolving assemblage of mate- 
rial, perception and transformation, Leibniz paves the way for understanding 
the pitfalls of cybernetic prediction as it is applied today. Such a critique is 
echoed in the work of contemporary scholars like Orit Halpern, and is ripe for 
continued critical examination. Perception, to Leibniz, is a complex calculus 
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between the representation of the object, the subject perceiving that object, and 
the discursive properties of that interaction. Yet, it is the discursive nature of 
communication systems that cyberneticians often fail to consider. As Halpern 
(2014) notes, Wiener understood that not all forms of information (e.g., meta- 
phorical representations, connotative meaning, denotative descriptions, etc.) 
could be recorded into cybernetic systems, thereby making the foundation of 
prediction recognised today wholly incomplete. 

In our contemporary context, and with an ever-increasing black-boxed 
world subsuming ontological truth, revisiting the theological investigation of 
the provenance of ‘natural’ or universal truth is necessary. As Sconce (2019) 
keenly observes, 


In this post-human universe of secular data management, the immate- 
riality of information replaces the ontological infinitude of God as the 
occult field of magical omniscience, promising its acolytes, through 
the transubstantiating miracle of magical positivism, the possibility of 
deliverance from the mortal humiliations of material existence. (235) 


This is not to say that a purely theological view of truth, or life itself, should 
be embraced. Rather, the same rigour of inquiry that has defined metaphysi- 
cal philosophy since Descartes must be applied to contemporary instantiations 
of often unquestioned truth: The black box, the algorithm, the cybernetic veil of 
AI. In his essay, The Technology of Enchantment and the Enchantment of 
Technology’, Alfred Gell (1992) urges such an approach by examining the pro- 
cess of creation. Creators, through imbued skill and cultivated craft, are often 
revered as gods amongst their human peers. Yet, when the artefact is a techno- 
logical system, the question emerges: Should we allow the unquestioned sov- 
ereignty of those who create the systems that ever-increasingly seek to orches- 
trate and prime our daily outcomes simply because those creators possess a 
skill we do not? What Gell advocates, perhaps unintentionally, is something 
that many - from philosophers of technology to everyday consumers - grapple 
with today: The godlike status those who create are granted, often passively, 
by those who rely upon the skilled to navigate an increasingly technologically 
dependent society. 

During an era wherein human and technological systems have become ever- 
more intertwined - often to the point of obscurity - a critical understanding 
of this godlike and unquestioned role humans play in developing technological 
systems is increasingly necessary; particularly as these systems have become 
the hidden blueprint of our sociological condition. As Wiener states: 


Those who would organize us according to permanent individual func- 
tions and permanent individual restrictions condemn the human race 
to move at much less than half-steam. They throw away nearly all our 
human possibilities and by limiting the modes in which we may adapt 


198 Al for Everyone? 


ourselves to future contingencies, they reduce our chances for a reason- 
ably long existence on this earth. (52) 


As is the case with software like PredPol, using broad categories of social data 
to predict individual behaviour not only misapplies cybernetic principles of 
learning vis-a-vis feedback beyond its narrowly defined parameters, but risks 
limiting human possibility as Wiener warned. Instead of fetishising algo- 
rithmic futures, researchers should continue the endeavour of philosophical 
questioning of algorithmic contingencies and the point of creation, as well as 
practical inquiry into the genesis of the data, how that is accrued, and implica- 
tions of relying upon categories to ‘predict’ individual action. 

We must actively and critically embrace that humans, not sublime or 
other godlike manifestations, are the creators of artefacts that mitigate our 
ontologies. The implications of this acknowledgment are philosophically far- 
reaching, upending a culturally-entrenched power dynamic between creator 
of technology and unquestioning consumer that persists even today - an era 
saturated with information, simulation and, ultimately, primed prediction. 


Notes 


' Short for ‘predictive policing; for which the program has its own definition: 
“The practice of identifying the times and locations where specific crimes 
are most likely to occur, then patrolling those areas to prevent those 
crimes from occurring (PredPol n.d.). 

Essentially, the ‘broken windows’ theory of policing argues that if minor 
offenses or criminal acts are left unattended, thus indicating a lack of regard, 
more serious criminal activity and ‘urban decay’ will follow; see Kelling and 
Wilson 1982. 

3 As Harcourt (2007) notes, the first risk assessment instrument was intro- 

duced in Illinois in the 1930s. 


i 
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CHAPTER 12 


Algorithmic Logic in Digital Capitalism 


Jernej A. Prodnik 


Introduction 


In recent years research in social sciences and related academic fields has 
attributed increased importance to algorithms and their impact on social rela- 
tions and our everyday lives. While algorithms are nothing particularly new 
and can be closely related to computing or even mathematics as such, debates 
have slowly but surely moved beyond the narrow confines of the so-called hard 
sciences. They are now taking centre stage when authors analyse topics such 
as political communication, electoral campaigning and mass micro-targeting 
of potential voters (Moore 2018; Vaidhyanathan 2018), automated trading in 
stock markets and various other types of financial transactions (Pasquale 2015, 
Ch. 4; MacKenzie 2017; 2018), or the impact of technological innovations on 
journalism (Diakopoulos 2019). Their influence is emphasised in healthcare, 
loan approvals, transportation, traffic-control, city urbanization, education, 
employment, policing, security and even military conflicts (Fisher 2020; Bridle 
2018; Moore 2018; Munn 2018; Mosco 2014). Critical analysis has dem- 
onstrated their impact in constructing ‘digital poorhouses, since they have 
become prominent in state administration and eligibility systems for poverty 
management (Eubanks 2017). It is also impossible to ignore them when consid- 
ering technologies forming the Internet of Things and cloud computing (Bunz 
2014; Mosco 2014), search engines, digital social networking platforms and 
various recommendation systems, or ranking, reputation and personalisation 
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tools aimed at tracking and controlling of behavioural patterns (Mager 2012; 
Gillespie 2014; Prodnik 2014; Kitchin 2017; Srnicek 2017; Fuchs 2019). 

This is to name only some of the most prominent issues that recent research 
has focused on, with many more aspects of our lives affected on a daily basis 
(Willson 2016). There seems to be little doubt algorithms now play one of the 
central roles in almost all spheres of society, from politics and economy to cul- 
ture and interpersonal relationships, subsequently raising various types of ethi- 
cal issues (Mittelstadt et al. 2016; Coeckelbergh 2020). 

In digital environments algorithms overlap and mutually influence each 
other, forming what can be considered layered algorithmic systems or ensem- 
bles of algorithms (cf. Kitchin 2017, 18-21). In this chapter I will not explain 
individual algorithms in an abstract manner, but rather focus on the key char- 
acteristics and social consequences of such ensembles of algorithms in their 
current hegemonic social form (for practical reasons I will simply refer to 
them as algorithms). This will hopefully shed some light on the reasons for 
their increasing social influence. 

All technologies are inevitably embedded in - and influenced by - the social 
context in which they are developed, so my analysis will consider ensembles of 
algorithms as part of the competitive and inherently unstable capitalist society 
(Streeck 2012), or to put it more narrowly, as part of digital capitalism (Fuchs 
and Mosco 2015; Fuchs 2019). My contribution therefore aims to provide some 
answers on how algorithms work in digital capitalism, what are the key rea- 
sons for this and what is their impact for society at large. Focusing on digital 
capitalism assumes a theoretical framework of the political economy of com- 
munication, which points at the power asymmetries in society in an overarch- 
ing manner, while taking on board the fact there is nothing ‘natural’ in these 
characteristics of algorithms. It also helps to move the analysis beyond abstract 
notions that have a limited explanatory value in specific historical contexts. 


Understanding Ensembles of Algorithms in Capitalism 


In contrast to many other topics there is a large degree of overlapping in how 
authors define algorithms. Bunz (2014, 7), for example, notes that an algorithm 
is ‘a set of rules to be followed by calculations. This definition does not dif- 
fer significantly from either Bucher’s (2017, 31), in which an algorithm ‘is just 
another term for those carefully planned instructions that follow a sequential 
order} or Kitchin’s (2017, 14), for whom algorithms are ‘sets of defined steps 
structured to process instructions/data to produce an output’ In this sense all 
computer software and digital technologies are fundamentally composed of 
algorithms (ibid.). Even though they were put forward by social scientists, such 
definitions are quite abstract and cannot explain by themselves why the social 
impact of algorithms has been so significant in recent years, especially since 
there is no inherent technical necessity for their increased omnipresence. 
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As already noted, my aim is not to look for universal characteristics of 
algorithms - even if that were possible or made sense in social sciences - but 
to understand them as part of the existing historical epoch, where they are 
bundled together in vast and overlapping digital ensembles, predominantly 
under the control of powerful capitalist corporations. Not to interpret them 
as technical or mathematical constructs, but through their social causes, pur- 
poses and consequences when implemented and executed (cf. Mittelstadt et al. 
2016, 2-3). This choice comes close to popular definitions of algorithms and 
has obvious downsides. It leaves much room for ambiguity and either risks 
making the scope of analysis too expansive, or puts too much focus solely on 
what could be called ‘mega-algorithms, while ignoring the more basic ones. It 
is exactly these algorithms, however, that are most influential and consequen- 
tial. As such, they must be subject to thorough scrutiny. 


Algorithms as Narrow Artificial Intelligence 


Before continuing I must note that for the purposes of this chapter I consider 
algorithms as part of a narrow form of artificial intelligence. They have lim- 
ited autonomy beyond the tasks which they were made for. While the so-called 
Artificial General Intelligence has the capacity to behave intelligently in a wide 
variety of contexts and use knowledge in novel situations, emulating intelli- 
gence of human beings, it remains in the realm of speculation (Boden, 2016; 
Dyer-Witheford et al. 2019, Ch. 1; Mitchell 2019, Ch. 3; Coeckelbergh 2020). 
What is sometimes called narrow AI, however, is already widely present and 
exists in our everyday lives. It can be connected to algorithmic processes that 
normally address narrow tasks, which means that their application cannot be 
generalised to other domains of functioning. State of the art AI still lacks real 
understanding and thus flexibility to operate outside the frontiers of their own 
design (ibid.). 

Because algorithms ‘don’t know what they don't know’ human beings have an 
advantage especially in complex communication, expert thinking, and creative 
tasks (Diakopoulos 2019, 29-30, 122; cf. Bunz 2014, 17; Mitchell 2019). It is 
also very challenging for computers to perform non-routine tasks, as human 
beings have large reservoirs of tacit and contextual knowledge, which they are 
not even aware of (so-called Polanyi’s paradox). The situation is similar with 
our most basic and unconscious sensorimotor abilities, including walking, 
manipulating objects or understanding complex language, which may be very 
simple tasks for human beings but are amongst the biggest challenges for engi- 
neers (Moravec’s paradox). These issues are currently generating considerable 
engineering bottlenecks (see Frey 2019, 233-236). 

The currently dominant paradigm in AI is machine learning, for example, via 
artificial neural networks which try to mimic human brains. Instead of being 
built top-down as a set of logical rules for handling data, machine learning 
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systems use an inductive approach for finding patterns, which are often based 
on statistical calculations and probability. A statistical pattern-recognition 
approach presupposes pattern extraction from data, with these systems creat- 
ing their own models of inference. Developed solutions are therefore based on 
the data itselfand on what these algorithms have previously learned (see Boden 
2016; Mitchell 2019; Dyer-Witheford et al. 2019, 8-15; Bridle 2018, Ch. 6). 

Ihe fact that machines now continuously learn on data means that actors 
and institutions, that have access to quantitatively more and/or qualitatively 
better information, are in an advantageous position. They can improve the 
quality, effectiveness and capacities of their algorithms. This is an important 
point I will return to when describing the characteristics of algorithms in digi- 
tal capitalism. Nonetheless, as a narrow form of AI these systems can currently 
generalise only on data they were trained for, and therefore merely simulate 
real intelligence. 


Embedding Algorithms in Capitalism 


To say that algorithms have to be considered as part of capitalist society may 
seem fairly inconsequential, as I noted at the start of this chapter. But this is 
a system with certain tendencies and basic characteristics that influence all 
phenomena operating within it. Even though these tendencies can be coun- 
tered or partially neutralised in many ways, most obviously through politically 
enacted regulation, they are the result of existing and dynamic social structures. 
They do not pre-determine the outcomes, but they do set the framework and 
delimit the level of possibilities within that system (cf. Collier 1994). In other 
words, capitalism has a specific logic in how it operates, and the impact of that 
logic can be identified and analysed in various phenomena that work within 
this system. 

A concise definition of capitalist society is provided by Streeck (2012), who 
argues that this ‘is a society that has instituted its economy in a capitalist man- 
ner, in that it has coupled its material provision to the private accumulation of 
capital, measured in units of money, through free contractual exchange. Simi- 
lar to social scientists in the 19th century, he emphasises that there cannot be 
any strict empirical separation between society and economy because of their 
interrelatedness. Furthermore, economic relations are constantly attempting 
to consume non-economic relations through commodification, since this is 
a system that needs to expand constantly, paradoxically staying stable only 
when being in movement (cf. Prodnik 2016). Competitiveness, permanent 
revolutionising - presupposing continuous change, innovations, instability 
and uncertainty - and expansion of capital are therefore part and parcel of this 
system and influence all social relations (Streeck 2012, 5-9). 

In a critical and holistic approach of the political economy of communication, 
it would thus not only be disadvantageous, but quite impossible to completely 
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dissemble algorithms from the wider capitalist context. Major ensembles of 
algorithms today are developed and owned by some of the biggest corporations 
in the world (Mosco 2014). Alphabet (Google), Facebook, Microsoft, Apple or 
Amazon might be seen only as tech companies, but they are expanding into 
and influencing numerous other branches of the economy and consequently 
our lives. Many other companies that primarily function as digital platforms, 
such as Uber or Airbnb, have brought similar economic disruption not only 
to the most prominent geographic locations, but also to peripheral ones (cf. 
Srnicek 2017). Even the automotive company Tesla views itself first and fore- 
most as an innovative tech company, conspicuously basing its headquarters in 
Silicon Valley. 

It is not only that algorithms play one of the most important roles in all of 
the cases mentioned above, they also clearly demonstrate it has become impos- 
sible to speak of ‘digital-only’ projects, that would somehow be separated from 
the non-digital world. In digital capitalism many formerly clear borders and 
demarcation lines have converged or completely collapsed as commodification 
seeps into every part of our lives, social practices and relations (Prodnik 2016). 

Even though major digital corporations are in many ways breaking new 
ground, they are not entirely dissimilar from corporations of the old. They 
are in perpetual quest of either short-term or long-term profits and new areas 
where they could expand to, while constantly struggling to innovate and 
increase their market share. These very basic pursuits largely delimit the man- 
ner in which they design algorithms and why they are developed in the first 
place (cf. Mager 2012; Gillespie 2014, 176-177). Bilić (2018) points out that this 
is one of the central reasons why algorithms cannot be seen simply as techni- 
cal artefacts. In the case of Alphabet, for example, algorithms are ‘also business 
strategies for market control and dominance’ (ibid. 71). This should be taken 
aboard before pondering further about the characteristics of algorithms, as it is 
relevant throughout the chapter. 


Characteristics of Algorithms and their Structural Reasons 


A study of the literature on algorithms cited in the previous sections makes it 
possible to define four basic characteristics of algorithms in digital capitalism: 
(1) opacity and obfuscation, (2) datafication, (3) automation, and (4) instru- 
mental rationalisation. There are both structural reasons for these character- 
istics as well as wider consequences they could have on social relations and 
social totality. While these characteristics can be analytically separated, they 
are thoroughly interconnected in practice and frequently reinforce each other. 
The key point of emphasis mentioned earlier is that these characteristics should 
not be seen as universally inherent to algorithms, since they are to a consider- 
able degree a product of the existing social order - digital capitalism. To put 
it differently, in a different political-economic context, there could be other 
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structural reasons at play, thus leading to changes in these basic characteristics 
or at least in their prominence. 


Opacity and Obfuscation 


Ihe first fundamental characteristic of algorithms is opacity and obfuscation, 
which is mainly an outcome of their secrecy and restrictiveness, but also of 
technological complexity and multiplicity. In essence, how algorithms actu- 
ally operate is to a large degree incomprehensible and difficult to understand, 
often even for experts. While we have basic ideas about the major algorithms, 
discerning details of how exactly they work, what data they are collecting, 
how it is used, why certain results finally appear, or who has access to them, 
is much more difficult or even impossible. Pasquale (2015), for example, notes 
that algorithms are secretive and restrictive black boxes. This seems like an 
apt metaphor, since it denotes both a recording device and ‘a system whose 
workings are mysterious; we can observe its inputs and outputs, but we cannot 
tell how one becomes the other’ (ibid. 3). Even though algorithms have wide- 
ranging consequences for the shape and direction of our societies, this means 
they are ‘opaque and inaccessible to outside critique; and their parameters, 
intent and assumptions indiscernible’ (Willson 2016, 4). 

There are three major structural reasons for this characteristic. Firstly, they 
are privately owned and subject to various types of intellectual property rights 
(copyright, patents, trademarks, etc.), that generally presume secrecy (ibid.). 
As emphasised by Pasquale (2015, 61), ‘the huge companies resist meaningful 
disclosure, and hide important decisions behind technology, and boilerplate 
contracts’ (cf. Kitchin 2017, 20). While most companies that own algorithms 
have obvious commercial reasons to keep them opaque - intellectual work, 
as part of them, can serve as an important market advantage when competing 
with other companies (cf. Bilić 2018) - making them completely transparent 
could also lead to security breaches and attempts of manipulation. An obvious 
example is the gaming of search engines by rogue websites. 

Even if there was full transparency on how algorithms work, most internet 
users would have serious problems if they tried to meaningfully comprehend 
them (Obar 2020; Willson 2016, 10). A digital divide can therefore be seen 
as the second structural reason for opacity, one that can be connected to the 
lack of expert digital literacy and programming knowledge of lay users. Power 
asymmetries and social inequalities are leading to exclusion in the digital as 
well as non-digital spheres as most people have vast difficulties with much 
more basic online understanding than complexities of algorithmic procedures. 
A poll conducted by the Pew Research Center (2018), for example, revealed 
that the majority of Facebook users had almost no knowledge of how their 
news feeds work. One can therefore only imagine how far from being able to 
understand the complexities of algorithms most internet users are. 
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Comprehending how algorithms operate, however, is not difficult only for 
the average user of the internet but even for experts. Several factors contrib- 
ute to this, including the fact that they are ‘always somewhat uncertain, provi- 
sional and messy fragile accomplishments, often worked on by large teams of 
programmers that constantly change and have a highly specialised division of 
labour between them, making an overview of the whole programming process 
difficult (Kitchin 2017, 18, 21; Bridle 2018, 40). In an analysis of financial algo- 
rithms, Pasquale (2015, 123; cf. 32), for instance, noticed how sometimes ‘black 
boxes are so effective they even ‘fool their creators’. This is the third structural 
reason for opacity, which can be closely connected to the fact we are speaking 
about large ensembles of layered algorithms that are interconnected, mutually 
influencing each other and constantly expanding, multiplying and changing 
(cf. Willson 2016). Furthermore, results of sub-symbolic AI systems, for exam- 
ple deep learning neural networks that are increasingly used in machine learn- 
ing, are very difficult to unpack, because they do not use symbols and a logic 
that is understandable to human beings (Mitchell 2019). 


Datafication 


Most algorithms make little sense or cannot even operate without the data 
which they process. Algorithmic decisions are made on this basis, meaning that 
the effectiveness of algorithms is ‘strongly related to the data sets they compute’ 
(Bunz 2014, 7). This is why, as I noted earlier, data is an increasingly important 
commodity in digital capitalism. 

There are, again, several reasons for this characteristic, the most obvious being 
that decisions made by algorithms are based on computational calculations that 
can usually be made only via quantifiable information. This inherent depend- 
ency on data has as a consequence a clear tendency towards the datafication 
of various practices and relations. Or to put it differently, the transformation of 
social reality and the world into structured data schemes that generally exclude 
nuance and wider context (Diakopoulos 2019, 117). Datafication should not be 
seen as something static, but as a continuous process; it is an important charac- 
teristic of major algorithms because they require constant flows of data (i.e. Big 
Data) to perform their key functions - a tendency so prominent because of the 
increased computing power and the near total ubiquity of digital networks and 
their tracking capacities (Prodnik 2014). 

It could certainly be argued that dependency on data holds true for algo- 
rithms as such, but it seems clear they truly gained relevance only with the 
availability of large troves of information that enable complex inferences, cor- 
relations and predictions drawn on a large scale (and beyond the scope and 
capacities of human beings). This also leads us to the second structural rea- 
son for datafication, one that is closely related to the first one: there is a con- 
stant need for enhanced capabilities and effective algorithms in a competitive 
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environment, at least if they are to produce improved results and operate bet- 
ter. Frey (2019, 304) notes that ‘data can justly be regarded as the new oil. Even 
though this illustration is overused and can at best be understood as a some- 
what faulty metaphor, it is true that ‘as big data gets bigger, algorithms get bet- 
ter’ (ibid.). With the development of machine learning, exposing algorithms 
to more examples leads to improvements in how they perform tasks (Mitchell 
2019, Ch. 6). 

Datafication has become so pronounced only with the development of digital 
capitalism, where the constant need for more and more data has become botha 
self-perpetuating cycle and one of the central factors in the production process. 
If we borrowed a phrase from the Marxist conceptual apparatus, we could say 
that the sum total of the forces of production in a specific historical context had 
to be developed to a certain level to make this a real possibility. Viewing this as 
a universal characteristic of algorithms would therefore be difficult, since they 
do not need - as a necessity - vast quantities of data to perform the most basic 
functions. It is only when they become crucial in the production process that 
this becomes the case. Institutions and actors employing algorithms typically 
do so because they want to predict large scale trends, patterns and risks or try to 
exert control, again pushing towards datafication to come closer to this objec- 
tive. This is closely related to the properties of digital capitalism and can be seen 
as the third structural reason for datafication (cf. Mosco 2014; Prodnik 2014). 


Automation 


Datafication is directly associated with the automation of processes, functions 
and decision-making. Automation is what makes algorithms so appealing in 
the first place, but it also ‘means that information included in the database must 
be rendered into data, formalized, so that algorithms can act on it automati- 
cally’ (Gillespie 2014, 170). Automation and datafication are therefore mutually 
intertwined. An attempt of automating decisions structurally pushes towards 
more datafication, with access to (more) data often enabling more intensive 
and extensive automation. 

Automation is a very discernible characteristic that makes algorithms into an 
interesting option for various actors and institutions. It enables them to ‘make 
high-quality decisions, and to do so very quickly and at scale’ (Diakopoulos 
2019, 19). This can lead both to a qualitative jump in acceleration of functions 
or procedures and their considerable scalability. Again, it may seem perfectly 
reasonable to claim this characteristic is universally inherent to algorithms and 
has little to do with digital capitalism; however there are three interrelated, but 
analytically distinguishable structural factors that counter this seemingly com- 
monsensical notion. 

First, algorithms and their capacities can bring increased competitive advan- 
tages to the companies employing them. Fractions of seconds, incomprehensible 
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to humans, can bring literally millions in financial trading or vast reductions 
in labour costs (Pasquale 2015, Ch. 4; MacKenzie 2017; Bridle 2018, 106-109). 

Second, existing resources can be put to a much better use and can help 
move decision-making process beyond the limitations that are inherent to 
human beings. Formerly laborious operations are simplified and made easy, 
often literally a click away. Today, it seems almost incomprehensible to imagine 
a manual harvesting of data, the indexing of the internet or non-automated 
searching, which would be performed solely by human beings and did not 
happen instantly. 

Third, and related to the previous reasons, increased overall efficiency is 
another enticing prospect for firms when applying algorithms. Attempts of 
automation have of course been a constituent part of the industrial capitalist 
society and there is a constant tendency on the part of capital to replace work- 
ers with machines and reduce labour costs (Dyer-Witheford et al. 2019, Ch. 1; 
cf. Marx 1867/1990). Algorithms thus present merely another step in that direc- 
tion, but quite possibly a qualitatively new one, since human beings increas- 
ingly have difficulties competing in cost, efliciency and speed with automated 
systems, leaving the door open for the automation of entire labour processes. 


Instrumental Rationalisation 


Development and application of technologies is highly dependent on the wider 
power relations, values and ideologies in society. While this is no place to go 
into the theoretical nitty gritty of it, most critical approaches today acknowl- 
edge this fact. The political economy of communication, for example, empha- 
sised the historical interrelation between the US military and industry in the 
development of ICTs, and how these technologies were remodelled to fit capi- 
talist social relations (Prodnik 2014; Dyer-Witheford et al. 2019, 3; Fuchs 2019). 
Even if we would disregard the long history of ICTs - which, after all, have to 
be seen as constitutive for the development of AI - development of algorithms 
is usually just a means to reach a very narrowly defined end. In other words, 
it is highly rationalised and instrumentalised. As an example, we can take the 
digital social media that critical authors view first and foremost as attention 
machines, aimed at catching and producing consumers. But as noted by Vaid- 
hyanathan (2018, 87), how they work goes beyond distraction and exhaustion. 
It also dehumanises users, since ‘it treats us each as means to a sale rather than 
as ends in ourselves. 

Algorithms cannot be seen merely as technical artefacts, because this would 
fail to explain their social role and influence - something I underscored earlier 
in the chapter. As stressed by Bili¢ (2018, 60), they must be seen as expres- 
sions of a specific technological rationality predominant in capitalism. They 
are embedded within it ‘as a mode of production, a specific form of capitalism- 
algorithmic capitalism (ibid.). Other kinds of technological rationalisations are 
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always possible, but in capitalism imperatives of this system are predominantly 
imposed on technologies. Examining how search engines are constructed, 
Mager (2012) for instance noticed that boundaries emerging from capitalist 
social relations were woven into the practicalities and the operation of algo- 
rithms behind them. This produced specific biases and altered the whole digital 
ecosystem, producing what she called algorithmic ideology. 

The capitalist logic can therefore be seen as the main structural reason for 
instrumental rationalisation, being one of the fundamental characteristics of 
algorithms (cf. Fuchs 2019, 59). This describes not only the central reason 
behind this characteristic, but in many ways also key reasons for opacity, data- 
fication and automation. For Fuchs (2009, 8), instrumental reason is ‘oriented 
on utility, profitableness, and productivity, with its objectives reduced to cost- 
benefit calculations. At least to a degree this is present in all characteristics 
delineated above, and all are therefore contributing to the intensification of 
instrumental rationalisation. 


The Algorithmic Logic and its Social Consequences 


It is possible to identify a range of conceivable consequences resulting from 
the four characteristics of algorithms. A schematic overview of structural rea- 
sons and their social consequences is provided in Table 12.1. The list of con- 
sequences is far from exhaustive and their relation to the characteristics may 
not be as direct as presented. It should, however, capture at least the essential 
features of what can be called algorithmic logic in digital capitalism. 

What I am describing here are tendencies that are real in the abstract, but can 
in practice be counteracted in various ways, hence forming potential counter- 
tendencies that would limit their actual social impact. Social struggles and pro- 
tests could for instance force governments into political measures that would 
lead to a shortening of the workday, which in turn could ease the pressure on 
unemployment; regulation could curb mass surveillance and data harvesting; 
court decisions could limit the dominant market position of certain corpora- 
tions and its platforms or put a stop to facial recognition and so on. Various 
countermeasures are to be expected, but they should not lead us to believe 
these tendencies were not ‘real’ or present in the first place (cf. Collier 1994). 


Incomprehensibility, Lack of Accountability 
and Preserving the Status Quo 


An important consequence of opacity and obfuscation of algorithms is their 
incomprehensibility for both lay users and often also for experts. In essence, 
these are secretive artefacts in more than one meaning of the word, since 
their complexity is an important and non-intentional contributing factor to 
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their secrecy (Coeckelbergh 2020, Ch. 8). It is often the case that we do 
not understand how algorithms function and interrelate, what exactly their 
operation encompasses, what impact they have on our lives and under what 
conditions this happens. This is why algorithms can lead to results and con- 
sequences that might not be intended in the first place and sometimes cannot 
even be adequately explained. 

There have been numerous cases of encoded biases in algorithms such as 
racist profiling or sexism (Bridle 2018, 142), which were a consequence of 
comparable biases historically existing in society. The poet Joy Buolamwini, for 
example, criticized them in a project AI, Aint I A Woman (www.notflawless.ai), 
which focused on grave failures of facial recognition when it came to black 
women. A myriad of such incidents demonstrates both that algorithms are far 
from neutral artefacts, a point I return to later, but also that even their designers 
in many cases have difficulties understanding why certain results materialised 
in the first place. In one of the more famous instances, Grindr was linked as a 
related application to an app which was aimed at finding sex offenders, revolt- 
ing the LGBT community. What is telling is that this and many other similar 
examples usually surprised designers of algorithms themselves. Increasingly 
sophisticated, extensive and complex algorithmic processes mean that ‘unin- 
tended and unanticipated consequences are an obvious, and will be an increas- 
ingly common, outcome’ (Willson 2016, 8). 

According to Pasquale (2015, 14) strategies of secrecy and obfuscation in 
algorithms are aimed at the consolidation of power and wealth. This cannot 
be seen as surprising, since applying intellectual property rights can bring the 
owners competitive advantages. Many authors have advocated for more trans- 
parency as a solution to the problem of algorithms being black boxed, which 
is a worthy cause. But making them transparent does not in itself bring any 
meaningful understanding of how they function (Willson 2016; Coeckelbergh 
2020, Ch. 8; Obar 2020). Since they are layered and complex systems, these 
properties represent difficulties even for experts, not to mention activist groups 
or regulators that would have the capacity to curtail them. Neither does the 
transparency of algorithms touch on an even graver problem - the commodifi- 
cation and privatisation of data. 

Social scientists have started warning about the dangers of algorithmic pro- 
cedures for democracy, especially when it comes to the influence of the big- 
gest digital social networking sites (Moore 2018; Vaidhyanathan 2018). This 
happened because nobody beyond their owners has a real oversight over how 
these algorithms are used, even though they have vast influence over the politi- 
cal process. This lack of accountability can be seen as a fundamental problem, 
because legitimation is at the core of all publicly relevant decisions in demo- 
cratic societies (cf. Coeckelbergh 2020, Ch. 10). Pasquale (2015, 16) goes even 
as far as to claim that ‘transactions that are too complex to explain to outsiders 
may well be too complex to be allowed to exist. In his opinion the informa- 
tion imbalances have gone too far, particularly since corporations that own 
algorithms have become the new sense-makers of our world. The Big Data they 
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collect brings big dangers, with even the smallest of oversights potentially cre- 
ating life-changing reclassifications in algorithmic decision-making processes 
(for examples see Eubanks 2017; Coeckelbergh 2020). 

What seems apparent, therefore, is that datafication in many ways helps 
to reproduce or even reinforce the status quo, and with it the existing power 
asymmetries and social inequalities. 


Mass Ubiquitous Surveillance in a World of Privatised Data 


It goes almost without saying that a logical consequence of the ever-present 
datafication is mass and ubiquitous surveillance, with severe breaches of pri- 
vacy as the final outcome. In the last two decades digital surveillance via vari- 
ous ICTs has practically become a norm, which led to a formation of a whole 
new research subfield with Surveillance studies. In 2013 this became an even 
more vigorously debated topic after the Snowden revelations. There is no need 
to repeat the main arguments of these debates, beyond the fact that digital sur- 
veillance opens the door for new ways of sorting, classifying, profiling, seg- 
regating and thus also discriminating people, which again reinforces existing 
inequalities and brings about new social disadvantages (see Prodnik 2014; 
Mosco 2014; Fuchs 2019). 

It is essential to underscore that data is not simply one of the resources in 
what Srnicek (2017) calls platform capitalism or what Fuchs (2019) defines as 
Big Data capitalism. It has become the resource for major companies, especially 
in the case of machine learning (Coeckelbergh 2020). This is why datafication - 
and correspondingly Big Data and mass surveillance - is not simply an optional 
thing. If you block surveillance the effectiveness of algorithms plummets and 
many of the existing business models start to collapse. Surveillance and privacy 
breaches are therefore a necessary part of the algorithmic logic in digital capi- 
talism. They are not a bug but a constituent feature that powers its development. 

A continuous push for datafication also brings about a highly unequal con- 
centration of the ownership of the data, which is syphoned off using digital 
surveillance (cf. Mosco 2014). These information inequalities are even more 
intensive than in the past, when Perelman (2002, 5) pointed out that ‘intellec- 
tual property rights have contributed to one of the most massive redistributions 
of wealth that has ever occurred’ He based this assessment on the fact they were 
owned almost exclusively by the rich and the powerful. Processes occurring 
with algorithmic datafication, however, are accentuating and intensifying this 
problem even further. 


Neutrality of Algorithms and their Naturalisation 


Various studies have attested to the fact that algorithms are far from neutral 
technical artefacts (Willson 2016, 9-10). This is both because human biases 
are present in their development and because they are created with certain 
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purposes in mind, for example ‘to create value and capital; to nudge behav- 
iour and structure preferences in a certain way’ (Kitchin 2017, 18). Who 
creates algorithms and with what underlying aims is far from irrelevant. Face- 
book’s algorithms, for instance, highly value content that arouses strong emo- 
tional reactions (Vaidhyanathan 2018), which was not a neutral engineering 
decision of its creators. While this may make Facebook into a powerful tool for 
motivation - but especially for grabbing users’ attention - it also means it ‘is a 
useless tool for deliberation’ (ibid. 132, 144). It mainly sparks shallow declara- 
tions and potentially destabilises democratic procedures. 

As noted by Diakopoulos (2019, 18) ‘the judgments that algorithms make are 
often baked in via explicit rules, definitions, or procedures that designers and 
coders articulate when creating the algorithms. They are of course neither neu- 
tral nor objective, but what is true is that ‘they will apply whatever value-laden 
rules they encode consistently’ (ibid.). This contributes to the illusion of their 
neutrality, even though it merely moves discrimination, prejudices, stigmatiza- 
tion and disadvantages upstream (Pasquale 2015, 35). 

How Google sorts its search results or how Facebook organises its news 
feed may seem self-evident and almost natural for their users, a normal order 
of how things stand, even though it was based on very real human decisions of 
how these platforms present and sort content. Many of our activities and prac- 
tices of course become naturalised when they become part of our everyday rou- 
tines and we accept them without necessarily questioning the power relations 
constitutive for them (Willson 2016, 2). It would indeed be impossible to live 
our lives if we always scrutinised every step we took, even the most mundane 
ones. However, this is not the only reason for naturalisation of algorithms; both 
datafication and automation are contributing to the fact that algorithmic deci- 
sions appear neutral. They are based on objective calculative procedures, which 
indeed have no intrinsic biases in themselves. This ‘mathematical, computa- 
tional and rational design, which is necessary for algorithms and is acquired 
through datafication, creates ‘an aura of universality of reason, an aura of calcu- 
lable, efficient and truthful solutions to given problems’ (Bili¢ 2018, 59). Since 
these decisions are simultaneously also automated, they obtain what could be 
called epistemic purity, and with it a halo of authority (Diakopoulos 2019, 118). 
This can be related to a phenomenon called automation bias, in which auto- 
mated procedures are perceived as more trustworthy than nonautomated ones 
or even our own experiences (Bridle 2018, 40). This is particularly true in case 
of ambiguous situations, since ‘automated information is clear and direct, and 
confounds the grey areas that muddle cognition’ (ibid.). 


Temporal and Spatial Changes 


Automation will also produce noticeable changes in temporal compression 
and the way space is (re)produced. When processes, decisions and functions 
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increasingly become automated, they also get accelerated. Especially in the case 
of intangibles, the level of acceleration facilitated by algorithms cannot be meas- 
ured only quantitatively. The change is primarily qualitative in nature, because 
it leads beyond limitations inherent to humans. The most obvious example is 
High-Frequency Algorithmic Trading in the financial markets, which is highly 
unstable and has been largely automated, with human traders becoming more 
or less obsolete. Decisions are now made in microseconds, leading to ‘one of 
the most dramatic increases in speed in recent times, going ‘beyond those per- 
ceptible by human beings’ (MacKenzie 2017, 55; cf. Pasquale 2015, 128-132; 
Wajcman 2015, 17-21). 

Nevertheless, acceleration in trading cannot be explained solely with tech- 
nological advances in algorithms. It was a result of carefully planned decisions 
at the time these algorithms were designed, with speed purposively at the core 
of how they function (see MacKenzie 2017). It would therefore be both theo- 
retically and empirically wrong to make a direct causal connection between 
acceleration and changes in technologies, as if the latter were constructed in a 
social vacuum. As emphasised by Wajcman (2015, 3), ‘temporal demands [...] 
are built into our devices by all-too-human schemes and desires. 

In Rosa's (2013) general theory of modernity, social acceleration is a constitu- 
tive and unavoidable part of modern societies, but technological acceleration 
is only one of the three dimensions in what he calls the acceleration-cycle. The 
other two are acceleration of social change and acceleration of the pace of life. 
Technological acceleration is indeed based on technological innovations like 
algorithms, with competition providing incentives for their development and 
adoption (what Rosa calls the external economic motor). However, in isola- 
tion, technological acceleration could not by itself lead to social acceleration. 
In most cases new technologies enable us to save time and should therefore - if 
anything - contribute to a general deceleration. It is only in relation to the other 
two dimensions and the fact we live in a competitive (capitalist) society that 
technological breakthroughs in fact lead to social acceleration (ibid.). 

Ina similar manner, algorithms may actually slow down the way certain sec- 
tors function. MacKenzie (2017, 57-58), for instance, discovered that work in 
the trading sector has slowed down considerably. It became much less hectic, 
but this was down to the fact that the work itself changed completely. It was not 
performed by human traders anymore, but by programmers that developed 
algorithms. Even with such contradictory examples, the general effect of the 
adoption of algorithms will almost surely be further social acceleration, in line 
with other similar technological advances. 

Algorithms are also changing public and private spaces, and how we perceive 
and interact with them (Mittelstadt et al. 2016, 1). Algorithms are at the core of 
smart cities, they are creating new knowledge about space, they are (re)direct- 
ing traffic, procuring navigation and rewriting how we understand certain geo- 
graphical locations (Fisher 2020). Alexa’s algorithms are, for example, reshaping 
how we live in our private homes, while Airbnb is fundamentally transforming 
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how people see their dwellings, simultaneously changing city geographies 
(Munn 2018). In essence, algorithms are already remodelling time and 
space configurations. 


(Un)employment and Automation 


Several studies are warning that the current pace of automation could have 
a serious impact on future unemployment and global labour markets. It is 
expected that a combination of algorithms, robotics and computers will 
increasingly make human labour redundant, even without development 
of Artificial General Intelligence (Coeckelbergh 2020, 136-144). There are 
many technical problems connected to automation, but they are slowly 
being overcome with machine learning and by making simple tasks even 
simpler. This solution was already used in factory automation during the 
industrial revolution, when previously unstructured tasks were subdivided 
and simplified. Whereas there is certainly a lot of unwarranted hype con- 
nected to algorithms and AI, a long history of technological innovations, 
identified already by Marx (1867/1990, 562-563), attests to capital's con- 
stant tendency to make labour superfluous through automation. As noted 
by Dyer-Witheford et al. (2019, 4) the ‘dismissal of automation as a “cha- 
rade” is deeply ahistorical. In the past, ‘capital has made people and indeed 
entire populations disposable’ 

A research paper by Frey and Osborne, published in 2013, for example, tried 
to estimate the probability of computerisation for 702 detailed occupations in 
which 97 per cent of the American workforce was employed at the time (Frey 
2019, 319). They estimated that nearly half of all employments were at risk, with 
low-income jobs that required lower education to perform hit the hardest (ibid. 
319-321). Frey (ibid. 322) analysed other studies and they concurred it was 
especially unskilled jobs that were most exposed to the risk of automation. A 
policy brief by OECD (2018) forecasted less drastic impact of automation, with 
14 per cent of the jobs in OECD countries highly automatable and 32 per cent 
facing substantial change in how they are done. But their analysis also warns 
that the tasks AI cannot do are rapidly shrinking, with some jobs becoming 
entirely redundant (ibid.). 

It is unlikely all occupational areas will go through such a radical transfor- 
mation in the mid-term as jobs in financial trading (MacKenzie 2017), but it 
seems that only a few will remain unaffected (Frey 2019, Pt. 5). While estimates 
regarding the proportion of occupations under direct threat remain specula- 
tive and vary because of differences in methodologies, it is highly doubtful they 
will all be offset by completely new occupations. Collins (2013) is amongst the 
authors that are convinced capitalist societies are facing the end of the middle- 
class work as we knew it because of technological displacement. He predicts 
even starker inequalities. Considering how deeply unequal societies today are, 
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and how uneven ownership of the algorithmic means of production is, we have 
every reason to be sceptical that the benefits of these processes will be evenly 
shared by the majority of the population. 


Conclusion: Algorithmic Necessity? 


Once it is formed, a system takes on a life of its own. 
- Haruki Murakami (1084) 


In a growing number of social domains decisions are influenced or directly 
made by algorithms. It remains to be seen how far reaching their influence 
will be in the long-run, but it seems increasingly likely that different corpo- 
rate actors and state institutions will either adopt algorithms or use them even 
more widely than they currently do. This tendency can be called algorithmic 
necessity, indicating that it is increasingly inevitable that different institutions 
will employ algorithms. Their adoption can have significant advantages on the 
market or can help to ‘rationalise’ administrative functions, which is always 
portrayed as a worthy cause in the neoliberal state. Non-adoption can similarly 
bring disadvantages, as companies that are incapable of innovation fall behind 
their competitors or simply fail to meet their quarterly goals. When one com- 
pany uses large quantities of personal data to improve their algorithms in an 
attempt to gain a competitive edge, others are likely to follow, which forms 
an almost self-propelling cycle. 

What Marx (1867/1990, 433) called ‘the coercive laws of competition, this 
iron cage of capitalist society, will therefore have direct influence on the general 
expansion of algorithms and how they are developed. Competition between 
different capitals that are structurally forced to constantly increase their accu- 
mulation, for example, pushes them into technological innovation (cf. Streeck 
2012, 5). With algorithms, this can lead to increases in productivity (preferably 
through automation), improvements in efficiency, or speeding up of the circu- 
lation of capital. As noted by Wajcman (2015, 17), ‘the faster that money can 
be turned into the production of goods and services, the greater the power of 
capital to expand or valorize itself. With capitalism, time is literally money, and 
"when time is money, then faster means better" and speed becomes an unques- 
tioned and unquestionable good. 

The mythological aspects of implementing technological innovations should 
not be overlooked either, even in the case when they might not be economically 
rational at all. It is easy to simply dismiss the hype surrounding technologi- 
cal breakthroughs, but in Mosco (2014, 5) view, such appraisals are mistaken: 
“The marketing hype supports myths that are taken seriously as storylines of 
our time. If successful, they become common sense, the bedrock of seemingly 
unchallengeable beliefs. Socially dominant myths acquire their own power and 
tend to become self-fulfilling prophecies. 


220 Al for Everyone? 


In digital capitalism the implementation of algorithms follows the logic of 
instrumental rationalisation that produces ‘irrational results’ and ‘impover- 
ishes human experience’ (Bili¢ 2018, 59-60). Authors of the Frankfurt School 
closely related instrumentalisation to the development of capitalism and the 
predominance of economic rationality in this system. They warned that inten- 
sification of these processes will lead to further social atomization, reification, 
domination and alienation. These are some of the most fundamental conse- 
quences of algorithms as artefacts of digital capitalism. 

These critical observations should not be taken as some Luddite rejection of 
technological progress, where the only path is either acceptance of algorithms 
or their complete rejection. Instead, there is no doubt that algorithms of a dif- 
ferent sort can serve democratic means, reduce human toil, reduce inequalities 
and help to bring about overall improvements in the quality of our lives. But 
this presupposes their fundamental reimagining in how they are made and for 
what purposes, together with political struggles that take into account the fact 
they can - and should - be changed if this is to happen. And this cannot be 
done without a change in who has control and ownership over these systems. 
In other words, this presupposes social relations that go beyond those imposed 
by digital capitalism. 
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CHAPTER 13 


'Not Ready for Prime Time': Biometrics 
and Biopolitics in the (Un)Making 
of California’s Facial Recognition Ban 


Asvatha Babu and Saif Shahin 


Introduction 


On 8 October 2019, Governor Gavin Newsom signed a law forbidding 
California police departments from using facial recognition (FR) software on 
body cameras. The decision was welcomed widely, especially by civil society 
groups that have long called for outlawing ‘an invasive and dangerous tracking 
technology that undermines our most fundamental civil liberties and human 
rights’ (ACLU 2019a). AB-1215, or The Body Camera Accountability Act, came 
on the heels of bans on government use of FR in five US cities earlier that sum- 
mer: San Francisco, Berkeley, and Oakland in California, and Cambridge and 
Somerville in Massachusetts (Cagle 2020). 

FR is a form of biometric artificial intelligence that involves ‘the automated 
process of comparing two images of faces to determine whether they represent 
the same individual (Garvie, Bedoya and Frankle 2016, 9). Attempts to use 
computers to identify human faces go back at least half a century (Goldstein, 
Harmon and Lesk 1971) and FR technology has become commonplace in 
recent years. We use it every day to unlock our mobile phones or tag friends 
on social media. Companies employ it to improve user profiles for targeted 
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advertising. Law enforcement agencies mostly rely on FR for two purposes: face 
verification to confirm a claimed identity and face identification to ‘identify an 
unknown face’ (Garvie, Bedoya and Frankle 2016, 10). 

Although both are contentious, it is the latter application that has raised the 
most eyebrows. Back in 2016, the Georgetown Law Center on Privacy & Tech- 
nology reported that FBI searches for identity using FR were ‘more common 
than federal court-ordered wiretaps’ (Garvie, Bedoya and Frankle 2016, 25). 
The faces of nearly 117 million Americans were already in federal law enforce- 
ment databases and every other American adult had had their photos searched 
in this manner. Law enforcement agencies in the United States have dabbled in 
FR-based surveillance projects since 2001 (Gates 2011). Now, with advances 
in technology, police departments in various cities in the US, in conjunction 
with technology corporations, have initiated FR-based surveillance programs 
that use existing infrastructure like CCTV cameras such as Detroit’s Project 
Greenlight, under which ‘cameras all over the city keep an eye on the popu- 
lace’ (Colaner 2020). Others are experimenting with using FR on body cameras 
and mobile phones (Naughton 2020). The US Customs and Border Protection 
agency began deploying FR cameras at airports in 2017 (Oliver 2019) while 
immigration officers started running FR searches on driving license photos to 
identify undocumented immigrants since at least 2019 (Chappell 2019). 

There is no federal law regulating the collection and use of biometric data in 
the United States. Illinois, Texas and Washington are the only states so far to 
pass comprehensive legislation regulating state and private collection of biom- 
etric information (Pope 2018). At the same time, facial data has become ubiqui- 
tous on the internet with the sharing of photos and videos on social media. An 
ecosystem of businesses, such as the controversial Clearview AI, have sprung 
up that take advantage of this ubiquity and regulatory lag to build massive data- 
bases and cheap tools for the state to use (Mann and Smith 2017; Naughton 
2020; Hill 2020; Kak 2020). 

Californias AB-1215 law, which came into effect in January 2020, is part of 
a slew of local, state and federal attempts to check this proliferation. In 2019, 
San Francisco, home to global tech giants and one of the most technologi- 
cally advanced cities in the world, became the first US city to ban FR use by 
law enforcement. It was quickly followed by Somerville, Massachusetts and 
Oakland, California (Metz 2019). In October 2019, California became the third 
state to issue a ban, following Oregon and New Hampshire (Thebault 2019). 
As the namesake of the Californian Ideology (Barbrook and Cameron 1996), 
an uncritical ‘technological solutionism’ (Morozov 2012) and belief in ‘dotcom 
neoliberalism, it is particularly interesting that California has been one of the 
earliest movers in attempting to regulate this emerging technology. 

In the wake of nationwide - indeed global - protests after the killing 
of 46-year-old George Floyd in May 2020 by the Minneapolis Police, reports of 
FR use by law enforcement to identify and arrest protestors has further fuelled 
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demands to remove the technology from the arsenal of law enforcement agen- 
cies around the country (Colaner 2020). In June 2020, the Detroit police came 
under fire for arresting an innocent Black man after FR technology flagged him 
as a shoplifting suspect. As activists mounted pressure on the city council to 
reject the proposed extension of the police FR contract, the Detroit Police Chief 
admitted that the FR system fails to accurately identify faces approximately 
95% of the time (Cameron 2020; Colaner 2020; Ferretti 2020). Around the 
same time, Boston became the biggest city on the US East Coast to ban FR for 
municipal use. Two US senators have proposed federal legislation calling for ‘a 
full stop to facial recognition use by the government at all levels nationwide 
(Ng 2020b). Even companies such as Amazon, IBM and Microsoft announced 
that they would not be selling the technology to law enforcement (NPR 2020). 

And yet, it may still be too early for critics of FR and other forms of algorith- 
mic surveillance to rejoice. Widespread calls to abolish carceral technologies 
and practices that disproportionately affect Black Americans (Benjamin 2016) 
have put pressure on private industry, not necessarily to stop building FR for 
the government but to at least manage the optics of their involvement. Seen 
in that light, some of these moves appear to be little more than short-lived 
attempts at corporate image management. Amazon, for instance, made it clear 
that its moratorium on the sale of its FR tool Rekognition to law enforcement 
would only be for a year. 

Without the visibility provided by these protests and the sustained pressure 
of activists, it is possible that some of these uses may have gone on unchecked 
and unexamined. Indeed, many of these companies continue to sell FR tech- 
nology to governments outside of the United States where there is little pres- 
sure from civil rights groups or movements like Black Lives Matter (Barik 
2020). Documents released a week after Microsoft’s announcement revealed 
it had previously been trying to sell FR technology to federal agencies without 
any regard for human rights, contrary to sentiments expressed by the com- 
pany’s president (Ng 2020a). 

Even AB-1215 may not be the ‘victory’ (Guariglia 2019) it seems at first 
glance. Initially intended to prohibit Californian polices use of FR on body 
cameras permanently, the bill was gradually diluted and defanged. The version 
that was passed into law proscribes FR use for only three years. In this chapter, 
we adopt a law and society approach - which views legislation as a social phe- 
nomenon (Ewick and Silbey 1998) - to explain how and why this came about. 
In this approach, researchers study law and legality as situated in society and 
laced through its culture by examining both the ‘formal and informal settings 
where legal activity - in all its guises - may unfold’ (Seron and Silbey 2004, 30). 
Our study traces the trajectory of AB-1215 from its introduction as a ‘spot bill’ 
on 21 February 2019 through its signing into law eight months later - both as 
legislative action within the state assembly and as public deliberation outside 
the corridors of the capitol. We critically examine legal documents as well as 


226 Al for Everyone? 


reports from the civil society and news media as a social discourse with a view 
to understanding (1) how FR technology was perceived and presented by dif- 
ferent stakeholders, specifically in terms of its benefits and harms and (2) what 
factors and actors contributed to the diminution of the law. 

We next discuss a range of socio-political concerns raised by FR and 
outline a conceptual framework for thinking about FR regulation in the 
light of these concerns. Then, following a brief discussion of the passage of 
AB-1215, we turn attention to our empirical analysis of the bill as a social 
discourse. We conclude with an assessment of how and why the bill was 
defanged and consider how it can inform future research and resistance 
against algorithmic governance. 


Inaccuracy and Mass Surveillance 


The application of FR for law enforcement raises three interrelated concerns. The 
first is the inaccuracy of the technology itself, or the possibility that an individ- 
ual is not who the FR algorithm identifies them as. That is because ‘biometric 
recognition is an inherently probabilistic endeavour’ (Pato and Millett 2010, 9). 
FR technology doesn't see faces - the way humans do - but offers probable 
matches based on geometric representations of facial features. Making comput- 
ers ‘see’ - i.e. engineering computer vision - is a difficult task that scientists and 
engineers have mulled over for decades. Advances in the ability to store 
and crunch large amounts of data as well as in machine learning algorithms 
have led to breakthroughs in recent years (Demush 2019). At first, the algo- 
rithm is trained to recognise which facial features are more likely than others 
to indicate similarity by analysing different images of the same person from a 
large training data set. This is known as machine learning. It then applies this 
learning to identify images of the same individual in real-life law-enforcement 
scenarios (Huang et al. 2008). But this machine learning involves ‘millions of 
variables’ - such as lighting, picture quality, and subject distance, for example 
- and is never perfect (Garvie, Bedoya and Frankle 2016). The FR algorithm, 
therefore, does not produce the ‘right’ match for an image but a series of more 
or less likely matches. 

In 2018, ACLU - the American Civil Liberties Union - tested the accuracy 
of Amazon's Rekognition, a top FR platform used by government agencies 
throughout the United States. The FR tool failed spectacularly: ‘the software 
incorrectly matched 28 members of Congress, identifying them as other peo- 
ple who have been arrested for a crime’ (Snow 2018). ACLU repeated the test 
in August 2019. This time, Rekognition misidentified 26 California legislators 
as criminals - among them Phil Ting, the author of AB-1215 (Chabria 2019). 
This lack of accuracy means that when law enforcement uses FR uncritically, 
many innocent people could end up on their radar, go to jail, be deported or, in 
countries such as the US, face the death penalty. 
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The second concern is their propensity for mass surveillance and violation 
of privacy. This happens in two ways. One, law enforcement agencies collect 
photographs of people to populate their FR databases - and they do so indis- 
criminately. Referring to the FBI program, the Georgetown Law report noted, 
‘Never before has federal law enforcement created a biometric database - or 
network of databases - that is primarily made up of law-abiding Americans’ 
(Garvie, Bedoya and Frankle 2016, 20, emphasis added). So far, these pho- 
tographs have been mined from people’s driving licences, ID cards and even 
social media accounts. As FR is employed via cameras at airports, on police 
personnel and so on, people will be subjected to facial ‘tracking from far away, 
in secret’ and en masse (10). This makes it nearly impossible for people to man- 
age their boundaries, a key practice in maintaining dynamic privacy relation- 
ships. This, in turn, makes it difficult to practice critical citizenship and aspire 
to the ideals of liberal democracy (Cohen 2012). 

Two, a significant feature of ‘high-dimensional’ - or individualised - data 
collection is that it allows cross referencing of multiple data sets (boyd and 
Crawford 2012). In other words, information about an individual in one data set 
can be used to find more about that individual by linking it with other data sets. 
So, while law enforcement agencies might compile FR databases putatively for 
identifying individuals in situations where a law is violated, they could eas- 
ily use the photographs to track various other activities of the individual - 
including activities that may be perfectly legal but politically undesirable 
for authorities, such as participating in protests against police violence. As 
was revealed by NSA whistle-blower Edward Snowden, this information can 
be easily obtained by the US government through social media and phone 
records (Greenwald, MacAskill and Poitras 2013). When this issue is coupled 
with FR’s inaccuracy, it means that ‘[e]ven if you're sitting at home on your 
couch, there’s a chance you could be arrested for protesting’ (Shwayder 2020). 


Biopolitics of Facial Recognition 


A third problem with FR systems is their proclivity to perpetuate and amplify 
social discrimination against marginalised communities. This is only partly 
a consequence of its technical flaws, which are also not arbitrary: ‘if a train- 
ing [data] set is skewed towards a certain race, the algorithm may be better at 
identifying members of that group as compared to individuals of other races’ 
(Garvie, Bedoya and Frankle 2016, 9). The Silicon Valley’s whiteness and appar- 
ent ‘colour blindness’ means FR training data are overwhelmingly White and 
therefore several times more likely to misidentify Black people (Buolamwini 
and Gebru 2018; Simonite 2019) - driving up their already disproportionate 
rates of incarceration. 

But social discrimination is not solely a function of machine learning or tech- 
nology design. Immigrants and minorities - Black, Latinx and Muslim people 
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in particular - are already much more likely than White people to be singled 
out for surveillance via FR at the level of policymaking (Bedoya 2019) and 
disproportionately represented in law enforcement and intelligence watchlists 
(Devereaux 2019). Browne (2010) argues that the practice of making a body 
visible - or ‘legible’ - has always been an exercise in power, with political and 
economic ramifications. The branding of enslaved Africans on American plan- 
tations, for instance, was a means of ‘accounting and of making the already 
hyper-visible body legible’ - not exactly the same, but also not altogether dif- 
ferent from the contemporary practice of making ‘bodies informationalized by 
way of biometric surveillance’ (139). Biometric technology reimagines the body 
as flows of data and patterns of communication (van der Ploeg 2002). Objecti- 
fied and digitized ‘individuals are broken down and reinterpreted in terms of the 
information provided by their body, instead of as agential social beings’ (Hood 
2020, 158). These data points are logged in a virtual register, making the bodies 
themselves legible, accountable, and thus controllable (Andrejevic, 2019). 

Considered from this perspective, FR is only the latest chapter in a long 
history of authorities using technology to subjugate, racialise and dehuman- 
ise people by acting upon their bodies - the newest arena of ‘biopolitics. A 
biopolitical view (Foucault 2003) brings to surface the systemic nature of social 
discrimination. In this view, the technological inaccuracy of FR is itself a 
consequence of institutionalised racism and classism - evident in everything 
from education to hiring practices to law enforcement - that keeps margin- 
alised bodies outside of Silicon Valley offices and inside of prisons. This view 
is at once micro and macro: it segues from the datafication of human body to 
map the geography of social belonging that such data enables. Gandy Jr. (1993) 
showed how the economic value of a person’s data differed depending on the 
social group they belonged to and how this differentiation reproduced social 
inequalities. As he noted, surveillance goes beyond social control and into the 
realm of sorting and differentially targeting people based on their positionality 
in the socio-economic hierarchy. 

Petit (2017) makes a distinction between discrete and systemic ‘externalities’ 
that accrue from artificial intelligence systems. Externalities could be either 
harms or benefits to third parties. Discrete externalities are ‘personal, random, 
rare or endurable’ (26). They take place at the level of the individual, may affect 
anyone with an equal chance, are low in frequency and neither ruin nor radi- 
cally improve the affected individual’s life. Examples include a malfunctioning 
robot mistaking a garden-variety rodent for a parasite and spraying it with 
pesticide. Systemic externalities are ‘local, predictable, frequent or unsustain- 
able’ (26). In other words, they are foreseeable, take place repeatedly, affect 
‘a non-trivial segment of the population’ and can cause a long-term ‘reduc- 
tion or increase in well-being of the local population class under considera- 
tion (26). For instance: China’s reported use of automation and cutting-edge 
technological tools to surveil and detain its largely Muslim Uighur population 
(Taddonio 2019). 
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This distinction helps us think normatively about AI regulation. Petit (2017) 
recommends that discrete externalities ‘should be left to the basic legal infra- 
structure’ (28). That is, emergent problems could be resolved on a case-by-case 
basis in an ex-post manner - or after they occur - by applying specific laws that 
are already in place. But systemic externalities require ex-ante consideration. 
As they are not only predictable but also significant in the scale of their impact, 
their repercussions need to be anticipated and lawmakers ought to institute 
regulations to mitigate the harms they might cause. 

Are FR’s externalities discrete or systemic? A purely technological view that is 
restricted to FR’s inaccuracy would consider them to be discrete - random, rare 
and occurring at the level of the individual. But a biopolitical view, as outlined 
above, enables us to see that FR’s externalities are in fact systemic in nature - 
causing frequent and permanent harms to large and identifiable populations. 
FR therefore requires ex-ante regulation that anticipates these harms and pre- 
vents them. 


A Brief History of AB-1215 


In a blogpost about ACLU’s 2018 test on Rekognition which demonstrated its 
fallibility, Jacob Snow, Technology and Civil Liberties Attorney with the ACLU, 
wrote: “These results demonstrate why Congress should join the ACLU in call- 
ing for a moratorium on law enforcement use of face surveillance’ (Snow 2018). 
This test from ACLU, combined with mounting academic research on the dis- 
crete and systemic harms of FR, has become a cornerstone for calls to ban the 
use of facial recognition technology by government agencies and law enforce- 
ment organisations (Metz 2019). AB-1215, in California, was one such call. 

AB-1215 was signed into law in October 2019 and came into effect in January 
2020 for a three-year period. It states that: ‘[a] law enforcement agency or law 
enforcement officer shall not install, activate, or use any biometric surveillance 
system in connection with an officer camera or data collected by an officer 
camera (Body Camera Accountability Act 2019e, 3). 

The spot bill for AB-1215 was introduced in February 2019 by Democratic 
member of Californias State Assembly, Phil Ting. With the support of ACLU of 
Northern California and several other civil society organisations, Ting intro- 
duced the first substantive draft of the bill in the State Assembly on 8 April. 
This version of the bill was considerably different from the final version that 
would eventually be passed. For instance, it prohibited the installation, acti- 
vation or use of biometric surveillance in connection with an officer camera 
indefinitely. It also made the law enforcement agency or official liable for dam- 
ages up to US$4000 in addition to attorneys’ fees (Body Camera Accountability 
Act 2019a). The bill made its arguments for a ban by drawing attention to the 
threat to civil liberties and the constitutional right to privacy and anonym- 
ity posed by FR, the possible chilling effect on free speech in public spaces, 
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FR’s lack of accuracy in identifying people of colour and women, and the 
disproportionate impact of this technology on overpoliced communities. 
The bill also repudiated law enforcement’s co-optation of tools meant to ensure 
their accountability (body cameras) into tools of dragnet surveillance (Body 
Camera Accountability Act 2019a). 

On 23 April, the bill was debated and amended in the Assembly Public Safety 
Committee with the recorded support of 24 civil society organisations (includ- 
ing the ACLU of Northern California) and opposition from law enforcement 
groups including the California Police Chiefs Association and California State 
Sheriffs’ Association. It passed through to the next stage with an amendment 
removing the specific amount that had to be paid in damages. When it passed 
with a 45-17 vote in the California State Assembly on 25 April, the ban on the 
use of FR in police body cameras was still indefinite (Body Camera Account- 
ability Act 2019b). 

In June, it was taken up in the California State Senate Committee on Pub- 
lic Safety, where it passed 5-2 without amendment. A couple of months later, 
while the bill was still on the backburner at the Senate, the ACLU of Northern 
California ran its second test using Rekognition which matched 26 Califor- 
nia lawmakers including Phil Ting with criminal mugshots (Gardiner 2019a). 
Almost two weeks later, on 27 August, the bill was amended to include a sunset 
clause that repealed the bill on 1 January 2027. The amendments also excluded 
from the ban ‘internal editing procedures for redaction purposes’; and the ‘law- 
ful use of mobile fingerprint scanning devices’ (Body Camera Accountability 
Act 2019c, 3-4). In September, at its final reading before passage, the bill was 
amended to shorten the sunset period from seven to just three years and was 
set to expire on 1 January 2023 (Body Camera Accountability Act 20194). 

Thus, when the bill was signed into law, it was not so much a ban on the 
use of FR by law enforcement as a relatively short moratorium on a very 
specific use of the technology. While this was still considered a win by civil 
rights groups around the country, ACLU and Assembly member Ting’s origi- 
nal intent to keep FR out of the arsenal of law enforcement in perpetuity was 
thwarted. How, and why, did this happen? And what can we learn from this 
qualified success to guide future attempts for regulating FR and other forms of 
algorithmic governance? 


Research Design 


To answer the questions above, we used the law and society approach (Seron 
and Silbey 2004) to track AB-1215 as social discourse, taking place both 
within and without the corridors of the state assembly, involving lawmak- 
ers as well as civil society groups, technology companies, police unions and 
the mass media. The passage of AB-1215 was a months-long process that 
involved these multiple stakeholders engaged in shaping the conversation 
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through reports, blog posts, press releases, media presence and participation 
in the legislative process. These artefacts offer important insight into the way 
the social discourse surrounding FR and its regulation evolved. Keeping in 
mind our objective to track this evolution, we conducted a qualitative content 
analysis of these artifacts. 

We triangulated data from three sources: legislative documents such as bill 
and floor analyses; communication materials from organisations listed in leg- 
islative documents as supporting or opposing the bill; and local news coverage 
on AB-1215 between February (the introduction of the spot bill) and October 
2019 (the signing of AB-1215 into act by the California governor). Although 
not exhaustive, these sources provided us with a comprehensive and diverse 
collection of stakeholders and their contributions to the conversation around 
AB-1215. They allowed us to examine the stakeholders primary participation 
in the discourse through their own publications and statements as well as what 
was picked up by the media. 

News reports were collected and combined from three databases - Access 
World News, LexisNexis and Factiva - using the search phrase: [‘ab 1215’ OR 
‘ab1215’ OR ‘ab-1215’ OR ‘a.b. 1215’ OR California AND (‘face recognition’ 
OR ‘facial recognition) AND (police OR ‘law enforcement’)]. Only articles 
published in the United States between 1 February 2019 and 31 October 2019, 
mentioning these terms in the headline or lead paragraphs were included in the 
corpus. Next, legislative documents were downloaded from the official Cali- 
fornia Legislative Information website. This included: the multiple versions of 
the bill from 21 February to 8 October (7 documents), bill and floor analyses at 
each stage (7 documents), and vote information. Finally, we manually down- 
loaded any communications material mentioning AB-1215’ from the websites 
of the civil society organisations mentioned as supporting and opposing the 
legislation. If there was no mention of the bill, we chose any materials that 
discussed ‘facial recognition’ within our target dates. This yielded a total of 38 
documents for analysis. After discarding duplicate and irrelevant articles from 
the media coverage, we had a total of 148 documents (96 news articles, 14 leg- 
islative documents and 38 outreach materials). 

Following Mayring (2004), analytical criteria were determined based on our 
research question as: arguments made in support of facial recognition use by 
law enforcement; arguments made against facial recognition use in body cam- 
eras by law enforcement; actors making these arguments; and social and politi- 
cal values present. These documents were then read line-by-line to develop 
inductive codes. In the first cycle of coding, we used the ‘in-vivo or ‘literal’ 
coding technique (Saldafia 2009) wherein exact phrases from the actual lan- 
guage of the documents can be used as code. According to Charmaz (2006), 
this technique helps preserve the meaning of actors’ views and actions within 
the code, making it easier to analyse while and after coding. Using this tech- 
nique, we developed a set of codes that were arguments for and against the 
use of facial recognition on police body cameras. We also noted the actors 
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making these arguments. In the second cycle of coding, similar in-vivo codes 
were grouped together under a summative ‘value’ to discern patterns in the 
data. Taebi et al. (2014), in their study of public values in technology and inno- 
vation, cite Talbot (2011) to suggest that the term refers to the public view of 
what may be considered valuable or worth striving for. 


AB-1215 as a Social Discourse 


The coding, categorization and organisation of data revealed that the main 
arguments for and against the use of facial recognition in police body cam- 
eras in California could be filed under three themes: (1) privacy, surveillance 
and liberty; (2) public safety; and (3) discrimination as technological artefact. 
Interestingly, both the pro- and anti-FR factions were able to use the concepts 
underlying these three themes to their advantage in their arguments. These 
themes are discussed below. 

Media discussions about the bill were dominated by civil society organisa- 
tions (especially ACLU), and lawmakers (especially Assembly member Phil 
Ting, the author AB-1215), followed by representatives of police unions and 
organisations. To a much lesser extent, comments and statements from tech- 
nology manufacturers such as Amazon, Axon and Microsoft were also fea- 
tured in news coverage. Most media coverage was local, with city and county 
news organisations contributing the bulk of the articles in the corpus. A lot 
of the coverage was based almost entirely on press statements from Assembly 
member Ting, congressional and senate floor analyses and press statements 
from organisations with involvement in the political process such as ACLU, 
Electronic Frontier Foundation, Fight for the Future, or, from the other side, 
the Riverside Sheriffs’ Association, the California Peace Officers’ Association 
and the California Police Chiefs Association. As a result, arguments tended 
to repeat themselves across the dataset. But as our corpus was multi-pronged, 
this enabled us to understand what arguments were more likely to be picked up 
from legislative debates and outreach materials and become popularised in the 
media - and what arguments were not. In addition, we were able to track how 
the discourse changed over time. 


Privacy, Surveillance and Liberty 


Fears that arming police body cameras with FR systems would lead to mass 
surveillance and intrusions of privacy were prominent in statements from 
civil rights groups and lawmakers supporting AB-1215, especially in the 
weeks and months following the introduction of the bill. For instance, Ting, 
the assembly member who wrote the bill, said on 9 May, when the bill went 
to the Senate: ‘Without my bill, face recognition technology can subject law- 
abiding citizens to perpetual police line-ups, as their every movement is 
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tracked without consent. Its use, if left unchecked, undermines public trust 
in government institutions and unduly intrudes on one’s constitutional right 
to privacy’ (Office of Assembly member Ting, 2019a). Similarly, ACLU rep- 
resentatives repeatedly brought up the ‘invasive’ nature of FR. Matt Cagle, an 
attorney for ACLU of Northern California, said, ‘AB-1215 helps ensure Cali- 
fornians don’t become test subjects for an invasive and dangerous tracking 
technology that undermines our most fundamental civil liberties and human 
rights’ (ACLU 2019a). 

The idea that FR would violate not some arbitrary notion of privacy but the 
constitutionally ordained rights of American citizens was important in these 
claims. The Fourth Amendment to the US Constitution protects citizens from 
unreasonable search and seizures. In effect, this allows citizens to be present in 
public without having to show any form of identification to authorities (Body 
Camera Accountability Act: Hearing 2019a). This protection would vanish 
with the widespread adoption and use of FR by law enforcement to scan civil- 
ians on the street. As described in the text of AB-1215, this is the ‘functional 
equivalent of requiring every person to show a personal photo identification 
card at all times in violation of recognized constitutional rights. (Body Camera 
Accountability Act: Hearing 2019a, 3). 

In addition, critics warned of the technology’s potential to ‘chill’ free speech 
in public spaces. The case of China was brought up as an example, such as in 
this news report: 


If there is one cautionary tale that surfaces in discussions of this tech- 
nology, it is the case of Chinas policing through an array of cameras 
equipped with facial recognition software of the Uighurs, a largely 
Muslim minority in the western part of the country. (della Cava, 2019) 


Pushback against these arguments was divided. Some proponents of FR in law 
enforcement, such as Ron Lawrence of the California Police Chiefs Association, 
asserted that privacy would be respected and technology won't be misused. 
‘Let me be clear, law enforcement respects and understands the importance 
of protecting a person’s right to privacy, he said. “We believe a person's privacy 
should not be violated unless that person is a threat to themselves or to others. 
We stand by this and will continue to do so in the future’ (Lawrence 2019). 
But others, such as the Riverside Sheriffs’ Association in its official opposing 
argument on the legislative floor, argued that citizens did not have a reasonable 
expectation of privacy in public. They also questioned why ‘civil libertarians’ 
were only concerned about privacy now and did not speak up for the privacy of 
law enforcement officers when an earlier law mandated the public disclosure 
of body camera video (Body Camera Accountability Act: Hearing 2019b). 

The contradictory claims of FR proponents serve to justify the fears of FR’s 
critics. Even if one takes the assurances of officers such as Lawrence at face 
value, there would be others in law enforcement who simply do not respect or 
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recognise people’s privacy - nor the basic rights guaranteed to them in the law 
they are claiming to enforce. 


Public Safety 


The most common argument proffered by proponents of FR in policing was 
that it would improve public safety. To do so, they cited the ‘success of FR 
in other states and countries. Two frequently quoted examples included: the 
reported use of FR to capture the perpetrator ofthe Capital Gazette shooting in 
Maryland in 2018 (della Cava 2019); and a supposed 60% reduction in carjack- 
ings in Detroit after the installation of a citywide FR system (Lawrence 2019). 

In addition, law enforcement groups such as the Riverside Sheriffs’ Associa- 
tion frequently brought up Californias plans to host mega events - including the 
annual Coachella Arts and Music Festival and the 2028 Summer Olympics - 
and the need to ensure public safety at these events. A ban on FR, they claimed, 
would signal the state’s inability to protect participants and visitors and could 
potentially mean the events would move elsewhere. This argument, first 
made in official comments to the legislature in opposition of AB-1215 (Body 
Camera Accountability Act: Hearing 2019b, 10), was picked up and amplified 
by media coverage. 

But critics of FR turned the argument on its head. They claimed that the 
technology would undermine rather than improve public safety - especially 
for minorities. If law enforcement officers were to police these communities 
while wearing FR-enhanced body cameras, members of the public would likely 
hesitate to interact with officers, even as victims or witnesses of crimes, for fear 
of having their faces caught on camera and stored in a database in perpetuity. 
This would make the job of law enforcement more difficult and also put these 
communities in greater danger (Body Camera Accountability Act 2019a). They 
also argued that public safety can be undermined by law enforcement officers 
suspecting or arresting innocent civilians. 


Discrimination as Technological Artefact 


Indeed, the disproportionately negative impact of FR on marginalised com- 
munities - minorities and immigrants - was a theme that surfaced in many 
different ways. Lawmakers and civil rights groups made this a key part of their 
argument against FR from the outset. In his 9 May statement when the bill went 
to the Senate, for instance, Ting noted that ‘AB-1215 is an important civil rights 
measure that will prevent exploitation of vulnerable communities’ (Office of 
Assembly member Ting 2019a). 

Albert Fox Cahn, who founded The Surveillance Technology Oversight 
Project at the New York-based Urban Justice Center, wrote in a commentary, 
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“There's something unbearable about thinking that our country’s largest invest- 
ment ever in police accountability’ - referring to body cameras - ‘would be 
turned into a weapon against the very communities of colour that it was sup- 
posed to protect’ (Cahn, 2019). Similarly, the Electronic Frontier Foundation 
noted that FR exacerbates historical biases born of, and contributing to, over- 
policing in Black and Latinx neighbourhoods’ (EFF 2019). 

Curiously, a few months into the bill’s passage, there was an important shift 
in this argument. Social discrimination moved from being a consequence of 
‘historical biases’ to being a function of the inaccuracy of the FR technology 
itself. The shift was especially evident after Rekognition failed the ACLU’s test 
on 13 August 2019, during which it misidentified 26 California lawmakers as 
criminals - many of them, including Ting, being people of colour. This tech- 
nological failure quickly became central to Ting’s statements against FR and 
justification for AB-1215. As he said after the experiment: “This experiment 
reinforces the fact that facial recognition software is not ready for prime time - 
let alone for use in body cameras worn by law enforcement’ (ACLU Northern 
California 2019). 

To be sure, FR’s inaccuracy when it came to identifying people of colour, had 
been known for long. ACLU had run a similar test a year earlier, with similar 
results. However, as the new experiment, and Ting’s mobilisation of its results 
to back his push for the bill, began to dominate the news, the discourse about 
social discrimination shifted subtly yet recognisably. Journalists increasingly 
began to associate FR’s potential for exacerbating marginalisation to what Ting 
called its ‘dangerous inaccuracies’ as a technological artefact (Office of Assem- 
bly member Ting 2019b). 

Ironically, scientific support for the arguments of FR’s detractors soon 
became the bill’s undoing - at least in terms of what it was initially intended to 
be. Amazon disputed the findings of the experiment, claiming the researchers 
purposefully used a lower confidence threshold than recommended (they used 
the default settings in the Rekognition software) (Gardiner 2019a). The Infor- 
mation Technology and Innovation Foundation argued FR was, in fact, more 
accurate than human recognition and so was an improvement to existing tech- 
niques in terms of accuracy (Information Technology and Innovation Founda- 
tion 2019). Even those who acknowledged that the ACLU experiment revealed 
a significant problem added that the problem, being technological, could - and 
eventually would - be resolved. Both Axon, one of the largest manufacturers of 
body cameras in the US, and Microsoft, which has its own FR software, agreed 
that FR had an accuracy and fairness problem - in its current state. As an Axon 
report said, ‘At the least, face recognition technology should not be deployed 
until the technology performs with far greater accuracy and performs equally 
well across races, ethnicities, genders and other identity groups’ (Sumagaysay 
2019). The idea that the main problem with FR was a technological flaw that 
needed to be and could be fixed given time became dominant. 
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February 21, 2019 


AB-1215 spot bill introduced in Assembly 


April 8, 2019 
AB-1215 read in Assembly 


April 23, 2019 
Passed in Assembly Committee 
on Public Safety hearing (7-1) 


April 25, 2019 

Amended in Assembly: 

to remove mention of specific 
damages amount mentioned. 


May 9, 2019 
Ordered to Senate in 45-17 vote 


June 11, 2019 
Passed (5-2) in Senate Committee 
on Public Safety hearing 


August 27, 2019 

Ban amended in Senate to expire 
on January 1, 2027; 

Excluded from ban: 

FR use for redacting footage; and 
use of fingerprint scanning device. 


September 6 , 2019 
Ban amended: expiry date 
advanced to January 1, 2023 


September 11 , 2019 
Passed in Senate (22-15) 


September 12 , 2019 
Concurred in Assembly (47-21) 


April 17, 2019 

Microsoft announces that it refused to sell FR to law 
enforcement because: "inherent bias in facial 
recognition - which is largely trained on white male 
faces - meant that it would be less accurate 
identifying women's and people from ethnic 
minorities’ faces” (Hamilton 2019). 


May 9, 2019 

Ting refers to FR in terms of: surveillance, “perpetual 
police line-up", consent, public trust, privacy, and civil 
rights (Office of Assemblymember Ting 2019a). 


June 17, 2019 

Ting's first mention of the readiness of FR (in the 
media). USA Today article quotes Ting’s concerns of 
the “not ready for prime time” nature of FR. (della 
Cava 2019) 


June 28, 2019 

Axon announces that it will not add FR to bodycams 
based on recommendations from its ethics 
committee. “...facial recognition is not yet reliable 
enough to be used on law enforcement's body-worn 
cameras.” (Sumagaysay 2019a) 


August 13, 2019 

ACLU conducts test of Rekognition’s accuracy with 
photos of California Lawmakers. Ting among 26 
misidentified as matches for criminal mugshots. 
Ting doubles down on rhetoric of FR’s lack of 
readiness for society based on its inaccuracy. 


September 6, 2019 

Riverside Sheriffs’ Association Head publishes 
op-ed arguing against claims of inaccuracy by the 
ACLU. 


October 8, 2019 
Approved by Governor Gavin Newsom 


Figure 13.1: A Brief History of AB-1215. 


This discursive shift came as a shot in the arm for FR proponents on the 
assembly floor, such as the Riverside Sheriffs’ Association. They had already 
been arguing that, as a world leader in technology development, California was 
not the kind of state that would ban a technology simply because it wasn't per- 
fect. ‘Could any of us imagine a statutory ban on Microsoft Office or Apple's 
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iOS until the software was able to be certified as 100% flawless?’ they had asked 
in their official comments to the legislature in June (Body Camera Account- 
ability Act: Hearing 2019b, 10). 

Two weeks after the 13 August experiment, the bill was amended to include a 
sunset clause (see Figure 13.1). The indefinite ban on FR was now set to expire 
after seven years. On 6 September, the expiry was further reduced to three 
years. Days after these amendments, the bill was passed in both houses - and 
signed by Governor Newsom a month later. 


Conclusion: Why Biopolitics Matters 


Our study has examined AB-1215 as a social discourse, adopting a law and soci- 
ety approach that views legislations as socially negotiated and focuses, among 
other things, on the ‘construction of meanings’ of the law and how it influ- 
ences the legal/regulatory process (Ewick and Silbey 1998). Our analysis leads 
us to two broad conclusions to guide future social research and social action 
vis-a-vis FR and algorithmic governance in general. Firstly, the discourse com- 
prised both ‘particular’ and ‘universal’ features - arguments and claims that 
were rooted in the political and cultural contexts in which they were made but 
also drew on ideas and concepts that transcended those contexts. For instance, 
the concern with FR systems infringing upon citizens’ privacy is universal, but 
in this discourse, this concern was hybridised with guarantees of personal free- 
dom that were specific to the US constitution. Meanwhile, FR’s proponents, 
while echoing universalist claims about enhancing public safety with the aid 
of technology, also mobilised Californias identity as a forward-looking and 
technology-friendly state - home to the Silicon Valley - to delegitimise calls 
for banning FR from law enforcement. A hybrid analytical lens that is sensitive 
to both universal and particular characteristics of FR as a social discourse is 
therefore vital for producing a nuanced picture. 

Secondly, while FR’s negative externalities - or the harms to ‘third parties’ it 
can cause (Petit 2017) - were initially perceived as systemic, they later came to 
be constructed in more discrete terms, albeit with certain systemic elements. 
Specifically, social discrimination - which FR was expected to reinforce - was 
discussed in terms of ‘historical biases’ at first but eventually became a function 
of machine learning-related inaccuracies of the technology itself. Crucially, this 
shift also implied that the harms it caused would be individualised and ran- 
dom, rather than affecting large sections of the populace in a predictable man- 
ner. The extent of the harm was still deemed unsustainable - wrongly identified 
individuals could end up being in prison or worse - and thus regulation was 
still warranted. But because the shortcoming was now perceived as technologi- 
cal, the need for regulation was supposed to be temporary: technology would, 
after all, improve - as technology is always expected to - and inaccuracies 
would reduce and eventually go away. 
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In theoretical terms, we witnessed the discourse transforming from biopo- 
litical to technological determinist - foregrounding technology as the cause 
underlying social phenomena, good or bad, and de-emphasising concerns 
about institutionalised racism and mass surveillance (see also, Gangadharan 
and Niklas 2019). Ironically, this shift was precipitated by a well-intentioned 
scientific experiment carried out by a civil rights group. It is possible the ACLU 
experiment helped the bill get past the assembly floor by making the harms FR 
can cause appear more concrete and measurable. However, it simultaneously 
reduced the concerns with FR to the level of the technology itself - an example 
of what Selbst et al. (2019) have called a failure of abstraction. The social dis- 
course lost its biopolitical magnitude - and so did the bill. 

Shahin (2019) has drawn attention to the theoretical significance of ‘critical 
junctures’ - emergent conditions in which a social discourse takes on a new 
direction without the principal stakeholders intending it to - in shaping regu- 
lations about technology. The ACLU experiment in August 2019 was such a 
critical juncture in the discourse about AB-1215. Even though ACLU’s own 
position on why FR had no place in law enforcement did not shift after the 
experiment - its press releases, for instance, retained their focus on systemic 
issues - both Assembly member Ting and the media latched on to Rekogni- 
tion’s manifest failure as a piece of technology. This quickly undermined the 
original intent of the bill. To be sure, other factors may have also played a role in 
the willingness shown by Ting and other supporters of the bill to accept a sun- 
set clause - twice - and change the character of the bill from a permanent ban 
on the use of FR in police body cameras to a three-year moratorium. But the 
noticeable change in the social discourse following the experiment, coinciding 
with changes in the bill itself, does indicate that the experiment weakened the 
bill even as it became instrumental for its passage. 

Our analysis is significant not only for future research on FR but also for 
future efforts to check algorithmic governance, legislative or otherwise, in the 
US and around the world. Firstly, it underlines the significance of a biopolitical 
approach to understanding - and resisting - algorithmic governance. That does 
not mean technological flaws are not important to point out. But those flaws are 
themselves the consequence rather than the cause of institutionalised racism: 
they don't produce but serve to re-produce discrimination and marginalisation. 

Research and resistance therefore need to press forward with an agenda in 
which FR and other forms of artificial intelligence are viewed as sociotechni- 
cal artefacts interpellated in relations of power - produced by them even as 
they serve to reproduce those relations. Moreover, a technologically determin- 
ist outlook, where activists focus only on the machine, its algorithms, input and 
output, and not as much on the social contexts of its design and use, is not only 
a failure of abstraction (Selbst et al. 2019) - it is also a failure of strategy. Blam- 
ing the technology alone might appear attractive in the short-term but, as our 
analysis indicates, it does not aid the long-term goal of regulating algorithmic 
governance as a means of achieving social justice. 
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Secondly, these relations of power increasingly have both universal and 
particular - or global and local - dimensions. Being sensitive to such hybrid- 
ity is important for research on and resistance to algorithmic governance in 
countries like the US, as our analysis indicates, but even more so in the Global 
South. That is partly because sociotechnical artefacts such as FR are constructed 
in North America and Western Europe, along with certain norms and practices 
of governance, and are often then ‘localized’ (Zaugg 2019) in the Global South 
amidst different forms of social hierarchy. 

Understanding these hybridised dynamics opens new avenues for research and 
resistance aimed at exposing and destabilising such hierarchies. For instance, 
how are algorithms trained to discriminate against people of colour implicated 
in the biopolitics of societies where the entire population is ‘of colour’? If algo- 
rithms for governance are coded and trained from scratch rather than copied 
and pasted, what kinds of power and norms of control do they reflect and repro- 
duce? Comparative research across countries, and empirical research focusing 
on specific countries and contexts of design and use of these technologies are 
important. As new legal instruments are developed to regulate this technology, 
collaborations between legal scholars and scholars of social science are key in 
understanding how we can negotiate these technologies as a society. 

In conclusion, we emphasise the main argument of our study. Scholars and 
activists have long been aware of the role of algorithms and artificial intelli- 
gence in marginalising minorities and immigrants and reinforcing relations of 
power (boyd and Crawford 2012; Eubanks 2018). Biometric technologies, such 
as FR, are particularly insidious examples as they can act at the level of both 
the ‘body’ and the ‘body politic. They reduce human beings into data points 
that may be stored, manipulated and controlled en masse (Browne 2010; Hood 
2020). At the same time, they enable forms of domination that are systemic in 
nature: they have a long history and they are institutionalised in a variety of 
social practices. Indeed, algorithms themselves represent one such social prac- 
tice. The prejudices they exhibit are a consequence of the systemic bias they are 
produced by - even as the algorithms help re-enact and reproduce that bias. 
Research on and resistance to algorithmic governance should, therefore, avoid 
the trap of technological determinism and not lose sight of the systemic nature 
of their subject matter - the biopolitics of discrimination and domination. 
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CHAPTER 14 


Beyond Mechanical Turk: The Work 
of Brazilians on Global AI Platforms 


Rafael Grohmann and Willian Fernandes Araújo 


Introduction 


‘Artificial Artificial Intelligence’, the slogan of Amazon Mechanical Turk (AMT), 
a global AI platform, is as ironic as the history of the 19th century Mechani- 
cal Turk itself. But it is less ironic than perverse to see that part of the profit of 
one of the owners of Cloud Empire (Couldry and Mejias 2019) is related to the 
crowd work of millions of workers around the world. 

The slogan also reveals the simplistic nature of the debates about the future 
of work and artificial intelligence (AI) that carry persistent representations of a 
‘general artificial intelligence’ driven by Hollywood imagery. These discourses 
point to visions of AI playing a key role in the full automation of work, whether 
in positive or negative frameworks (Dyer- Witheford, Kjosen and Steinhoff 2019). 

These narratives make inequalities and ghost work (Gray and Suri 2019) 
invisible, even though crowdsourced labour performed by humans, in fact, 
supports AI. Thus, there are tasks that might, in theory, be performed by AI, 
but are cheaper and/or quicker to simply outsource to human workers’ (Wood- 
cock and Graham 2019, 58). However, AMT workers are not the only ones 
supporting AI. Companies like Appen, Lionbridge, Mighty AI, Clickworker 
and Spare5 also play a key role as data trainers for AI, with a variety of work 
activities on their platforms, including data training for self-driving cars. The 
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discourses of these platforms propagate meanings of future and progress. 
Appen’s slogan, for example, is: ‘confidence to deploy AI with world-class train- 
ing data - artificial intelligence will improve the world. This helps to consoli- 
date images of digital labour, AI and data among workers (Soriano and Cabanes 
2019; Beer 2019). 

Work behind artificial intelligence is called ‘ghost work (Gray and Suri 2019), 
‘clickwork’ (Casilli 2019) and ‘micro-work’ (Tubaro and Casilli 2019; Lehdon- 
virta 2016). These metaphors are attempts to name work activities on AI plat- 
forms. They are not definitive notions, but just illustrations. For instance, the 
fact that this work consists of individual, compartmentalised ‘tasks’ lasting per- 
haps only seconds or minutes, does not make it ‘micro. In a similar way, these 
workers do more than just click on ads. On the one hand, the multiplicity of 
tasks involves audio transcriptions and translations, describing images, record- 
ing videos and photos, and so on. On the other hand, we understand that work 
activities, whatever they may be, involve the entire (material) body of workers 
(Huws 2014). 

Whatever the name, these people work for global artificial intelligence plat- 
forms. The global character of these systems points towards an important factor 
in the complexification of the human work behind AI: the platforms’ geopoliti- 
cal dimension. Fuchs and Sandoval (2014) point to a new international division 
of labour (NIDL). However, work on global AI platforms is different from the 
circuit of labour involving the iPhone, for example (Qiu, Gregg and Crawford 
2014). On the global AI platforms there is no division between lithium battery 
production in one place and software production in another, although they 
depend, in a sense, on this circuit of labour and these digital infrastructures. 
In this case, there are a few companies from the Global North managing and 
controlling a crowd of workers from many countries in the world, mainly from 
the so-called Global South. 

Crawford and Joler (2018) show connections among human labour, data and 
planetary resources, including the production of data for AI as a circuit of digi- 
tal labour (Qiu, Gregg and Crawford 2014): physical labour of mine workers, 
labour in distribution centres, crowdsourced labour on global AI platforms, 
and so on (Crawford and Joler 2018). This means highlighting the material 
dimensions of the work behind AI, from workers’ ‘human intelligence’ to media 
materialities, digital infrastructures, with diverse impacts, including geological 
(Parikka 2015; Milan 2018; Murdock 2018). 

We agree with the notion of the ‘planetary labour market’ (Graham and 
Anwar 2019). Global AI platforms do not eliminate physical spaces and are 
dependent on material infrastructures. The planetary scale of these platforms 
means that ‘labour markets help clients operate unboundedly and trans- 
spatially, and allow them to reconfigure the geography of their production 
networks for almost zero cost’ (Graham and Anwar 2019, 28). The workers, 
although they ‘can sell their labour power globally, [they] are tethered to the 
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locales in which they go to bed every night’ (Graham and Anwar 2019, 29). 
According to the Online Labour Index (OLI) of the University of Oxford, the 
country of origin of the largest number of online freelance tasks is the United 
States, and the country with the largest number of online workers is India. 

Thus, understanding the inequalities in work for global AI platforms means 
going beyond AMT and workers from the Global North. In a digital economy, 
there are inequalities involving local workers and global platforms. There is no 
digital labour universalism or a homogeneous and unique notion of global 
workforce. There are, in fact, diverse work and AI scenarios around the world. 

Ihe aim of this chapter is to analyse the work of Brazilians on global AI 
platforms, mainly Appen and Lionbridge, from the workers’ point of view, 
understanding platform labour and the platformization of labour as a secret 
ingredient in automation and contextualising platform labour in terms of the 
Global South (Grohmann and Qiu 2020). The Latin American scenario is less 
well known in academic research on labour and Al in the Global South in rela- 
tion to countries like the Philippines and India. The focus of this research in 
Brazil is not intended to argue a singularity of this country, but to show the 
existence of other realities beyond Global North that are also invisible academi- 
cally from critical research on AI. 

In the following section, we discuss platformization of labour and AI. Then, 
we present the methodology, with interviews and a survey with workers, 
together with observations of Facebook and WhatsApp groups. The topics of 
analysis are: communication among workers, difficulties of work, lack of infra- 
structural reliability, workers’ strategies, definition of work and understanding 
of AI. The results reveal the complexity of working on global AI platforms and 
Al imaginaries. 


Platformization of Labour and Artificial Intelligence 


There is no AI without ‘ghost work (Gray and Suri 2019). Conversely, the 
work that supports AI is only possible with the existence of digital platforms. 
‘Platform’ is the term that refers to the sociotechnical infrastructures of digital 
conglomerates and that connect different parts - State, companies, consumers, 
workers, and so on (Srnicek 2016). In the literature on digital media, it high- 
lights the performative character of these structures (Introna 2016). Online 
platforms are considered mediators that not only enable or facilitate certain 
practices, but also actively shape, transform, and distort content, relationships, 
understandings, etc. (Gillespie 2014). Simultaneously, that contemporary 
sociotechnical model for organising practices inserts processes driven by algo- 
rithms and digital data into different contexts (Van Dijck, Poell and De Waal 
2018). The algorithm-driven logic that underpins these platforms is the conflu- 
ence of different factors - business models, user data, algorithms, data centres, 
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servers, etc. Thus, digital platforms are infrastructures that depend on data and 
algorithms and which present values and standards inherent in their designs. 

Van Doorn and Badger (2020), for instance, highlight the role of data assets 
and meta-platforms in platform labour. However, as Srnicek (2016) argues, 
there is not just one type of platform. The role of algorithms and data in 
platform labour depends on its mechanisms and ways of extracting value 
(Sadowski 2020). In the case of AI, platforms are the places for many workers 
to produce and circulate data to automate processes, that is, towards AI (Beer 
2016, 2019). Ghost work behind AI is a way of mining workers’ process for data 
(Neff, McGrath and Prakash 2020). Thus, working for global AI platforms is a 
type of data labour for automation. 

There is no platform without moderation, as state Gillespie (2018) and 
Roberts (2019). More recently, Gillespie (2020) highlights the promise of ‘the 
promise of AT and the question of scale regarding data. “The claim that moder- 
ation at scale requires Al is a discursive justification for putting certain specific 
articulations into place - like hiring more human moderators, so as to produce 
training data, so as to later replace those moderators with AT (Gillespie 2020, 2). 
This means understanding the heteromation of labour (Ekbia and Nardi 2017) 
on global AI platforms. 

The platformization of labour, in line with Casilli and Posada (2019) and 
Van Dijck, Poell and De Waal (2018), is linked to the growing dependence on 
digital platforms to get and/or maintain work activities. This process is a social 
synthesis of others: datafication, financialisation and neoliberal rationality 
(Sadowski 2020, Dardot and Laval 2013). Platform labour requires large-scale 
data extraction and collection for its surveillance and algorithmic manage- 
ment mechanisms to be successful. Sadowski (2020) states that there are three 
key mechanisms of rentier capitalism platforms: data extraction, digital enclosure 
and capital convergence. Moreover, the data is a form of capital for platform 
companies, expropriating and colonising workers’ resources, especially in the 
Global South (Couldry and Mejias 2019). 

The platformization of labour does not materialise in the same way for dif- 
ferent actors and social institutions. There is a heterogeneity of workers with 
relation to gender, race, class and geography issues. This heterogeneity means 
not only differences, but also inequalities that structure the platform labour 
(Van Doorn 2017, Grohmann and Qiu 2020). According to Abilio (2020), plat- 
form labour is generalisation and appropriation of the livelihoods of peripheral 
populations by the platform companies. 

In Brazil - the focus of this chapter - gig work is not an exception, but his- 
torically the way of life of most workers, an ordinary state. The change is that 
gigs are now platformized. According to Grohmann and Qiu (2020, 5), ‘analys- 
ing platform labour in the South means that patterns in the North are often 
erroneously assumed to have also existed in Latin America, Africa and Asia’s 
developing regions, as if labour precarity is a novel phenomenon. Thus, the 
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theme of regulation in platform labour is very different in a Latin American 
context, as workers generally do not want to become regular employees (Abilio 
2020). Platform couriers and drivers, for instance, want to have the feeling of 
autonomy, flexibility and self-management of their own work, which makes it a 
challenge to think about the regulation of platform labour. This does not mean, 
however, that they are not organising themselves into unions, associations and 
strikes (Grohmann et al. 2020; Grohmann and Alves 2020). 

The platformization of labour intersects social and geographical contexts 
with issues around platform materialities and design. Global AI platforms 
work differently in relation to companies like Uber or Deliveroo, although they 
have similar mechanisms, such as algorithmic management and surveillance 
(Woodcock and Graham 2019). First, companies and workers can be located 
in different parts of the world, and workers often work from their own homes. 
But this does not necessarily mean that the tasks performed by workers are 
global. Sometimes they are located in the workers neighbourhood, city or 
country, such as advertising analysis or text translation tasks. Second, payment 
methods vary. AMT, for example, only pays US and Indian workers in cash. 
In Brazil, workers receive Amazon store credit. Other companies, like Appen 
and Lionbridge, pay workers in dollars, which makes workers see themselves as 
part of ‘world-class work’ (Soriano and Cabanes 2019). This means that there is 
no homogeneity in practices across global AI platforms. According to Tubaro, 
Casilli and Coville (2020, 2), ‘some platforms such as Mechanical Turk cater 
to a diverse range of corporate needs, while others specialise in AI services: 
Thirdly, there are different worker perceptions regarding the platforms’ objec- 
tives. At Deliveroo, for example, the courier knows that he/she will deliver food 
from a restaurant to someone. At AMT or Appen, as our analysis shows, there 
is production and circulation of ideas about what it means to train algorithms 
and ‘work for AT. In other words, this can mean alienation from the circuit of 
labour in global chains. 

Global AI platforms accelerate the platformization of labour from the process 
of ‘taskification of labour’. The distribution of micro tasks to the crowd workers 
materialises from the data labour. According to Casilli and Posada (2019, 10), 
‘the standardization and the fragmentation of previously complex and special- 
ised processes are essential to run a platform ecosystem where the activities of 
users fit in and are synchronised with others: There is data circulation (espe- 
cially the so-called ‘good data’) only with the circulation of labour on global 
AI platforms (Beer 2016). According to Tubaro, Casilli and Coville (2020, 4), 
‘AI companies depend heavily on data resources, including not only raw data 
but also annotations that add extra meaning by associating each data point, 
such as an image, with relevant attribute tags’ 

Dyer-Witheford, Kjosen and Steinhoff (2019) state that Al is the current gen- 
eral condition of production, configuring an AI capitalism. However, this means 
neither general AI nor full automation. Platform labour is the secret ingredient 
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of automation (Casilli 2019). The work on global AI platforms symbolises the 
‘heteromation of labour, according to Ekbia and Nardi (2017). Heteromation 
means keeping human beings in the system, because capitalism needs human 
beings, extracting value in invisible ways, with new logics of wealth accumula- 
tion. According to Ekbia and Nardi (2017), humans are doing a lot of work 
and machines are getting credit. The role of human beings at work is invisible, 
although the companies’ discourse is towards valuing human beings: ‘platforms 
tell clients that human contribution has value, but not who these humans are 
and in what conditions they work’ (Tubaro, Casilli and Coville 2020, 10). 

The intersection of platform labour and heteromation is the synthesis that 
the future of work will not be exactly automation, but the growing taskification 
of labour. According to Tubaro, Casilli and Coville (2020, 2), ‘automation is still 
in the making and has not yet been deployed at large scale, [but] its demand for 
micro-tasks is already transforming the daily practices, experiences and career 
trajectories of thousands of workers worldwide’. This occurs, according to the 
authors, in processes of AI preparation, AI verification and AI impersonation. 
However, it is necessary to highlight that the ‘taskification of labour’ would not 
be something original or new, since the ‘salary per piece’ was already a reality 
for Marx (1894). 

Research leaded by Antonio Casilli and his group has shown that AI 
platforms can have multiple configurations, from local start-ups to global 
companies - and this is the emphasis of this chapter. Tubaro, Casilli and 
Coville (2020) point out that there are platforms like AMT and Clickworker 
whose workers perform tasks in the most diverse areas and there are other 
platforms more specialised, ones such as Spare5 and Mighty AI (now owned 
by Uber), focused on data training for self-driving cars. Thus, the multiplicity 
of possible tasks on AI platforms shows the flexible nature of their workforce. 

Most research on global AI platforms focuses on the Global North, such 
as Irani (2015), Milland (2017), Gray and Suri (2019) and Ludec, Tubaro and 
Casilli (2019) highlighting countries such as the United States and France, with 
a centrality of AMT. However, research by Ludec, Tubaro and Casilli (2019) 
reveals interesting data for research on global AI platforms outside United 
States. In France, there are about 500,000 workers registered in AMT. This is a 
much smaller number than other platforms like ClixSense (7,000,000 workers), 
Microworkers (1,215,829), Clickworker (1,200,000) and Appen (1,000,000). 
This reveals the impossibility of generalising the localised experience of AMT 
workers in countries like the United States. As we stated, there is no digital 
labour universalism. 

One of the few studies that veer away from that Global North trend is 
Graham, Hjorth and Lehdonvirta (2017) which focuses on Africa, but was 
written by Global North authors and does not focus exclusively on global AI 
platforms. Schmidt (2019), in his research on workers that train AI for self- 
driving cars, finds that most of the workers are from Venezuela. There is even 
a Brazilian in his research sample. Schmidt's work, nonetheless, does not go 
further into Latin America itself. 
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In Brazil, Kalil (2019), and Moreschi, Pereira and Cozman (2020) researched 
Brazilians working at AMT. Kalil (2019) interviewed 52 people, usually single 
and graduate men about 30 years old. The alleged reason for work in these plat- 
forms is the need for additional income. Kalil (2019) also investigated workers 
from the United States (sample = 685) and India (sample = 125). Some of the 
workers’ statements are: ‘too much work, too little pay, too much exploitation’ 
(Kalil 2019, 189) and ‘I can't earn the equivalent of a minimum wage even if I 
work more than eight hours a day’ (Kalil 2019, 189). 

The research by Moreschi, Pereira and Cozman (2020) presents survey results 
with 149 Brazilians working at AMT and observation in a WhatsApp group. 
The profile is similar to the findings of Kalil (2019): White men and 29 years 
old. They have been formally unemployed for a long time. In addition, they cite 
other global AI platforms for which they work, such as Clickworker, Appen and 
Figure Eight. However, the research focuses only on AMT. 

The authors consider the working conditions of Brazilians worse than that 
of countries like India due to factors such as ‘the role of AMT in Brazilian 
turkers’ economic lives, the consequences of the lack of direct payment and 
the importance of WhatsApp for organising’ (Moreschi, Pereira and Cozman 
2020, 61). This survey reveals that work of turkers is closely intertwined with 
the historical informality of labour in the country, a gig economy that existed 
prior to digital labour itself. ‘As Amazon does not make a transfer to their bank 
account, like turkers in some other countries can [sic], the turkers in Brazil 
find themselves at the bottom of an unregulated market’ Moreschi, Pereira and 
Cozman (2020, 61). 

Research on Brazilians in AMT reinforces that communication also supports 
the organisation of workers, although still in informal solidarities (Soriano and 
Cabanes 2019). In the WhatsApp group, workers help each other and present 
a ‘rhetoric that blends entrepreneurship with elements of religiosity and self- 
help’ (Moreschi, Pereira and Cozman 2020, 53). One of the research statements 
reveals that workers do not want to be a ‘ghost’: ‘I exist and I want you and oth- 
ers to know that’ (Moreschi, Pereira and Cozman 2020, 47). 

In general, the literature review shows that, on the one hand, there are connec- 
tions between the tasks of workers for AI platforms in many parts of the world, 
with the potential for circulation of workers’ struggles (Dyer-Witheford 2015; 
Englert, Woodcock and Cant 2020). On the other hand, there are specificities - 
of the Global South and, in this case, of Brazil - in relation to payment, task 
supply, working conditions, difficulties in accessing the platform and prob- 
lems with language. The background of workers is also a central difference 
due to issues such as training and the legacy of the informal economy in 
the country. 

Thus, this chapter aims to analyse something not yet explored by research: 
how Brazilians work on other global AI platforms such as Appen and Lion- 
bridge, in order to highlight other inequalities involving AI and labour. From 
the class composition perspective (Woodcock 2019; Cant 2019), we focus here 
on the technical composition of the workers. This does not mean to disregard 
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the social and political dimensions of class struggles, but to focus the discus- 
sion on work activities. 


Methodology 


From February to April of 2020, we conducted exploratory research across 
groups where Brazilians discuss their work for global AI platforms. We under- 
stand exploratory research as a set of methodological practices developed to 
offer ‘new and innovative ways to analyse reality’ (Reiter 2017, 131). To this 
purpose, we provide transparent guidelines about the research process, aiming 
to demonstrate its reliability and validity (Reiter 2017). Based on this methodo- 
logical framework, we began with a preliminary online search for content on 
the subject, from Brazilian blogs and YouTube channels on this topic. Regard- 
ing this content, and considering the previous studies in the literature about 
digital labour in Brazil (Moreschi, Pereira and Cozman 2020; Abilio 2020; 
Grohmann and Qiu 2020), we were able to create the following list of AI plat- 
forms beyond Mechanical Turk: Appen, Lionbridge, Clickworker, MightyAI, 
Clixsense, Pactera, iSoftStone and Streetbees. 

Starting from this point, we investigated these platforms on LinkedIn, a 
social networking platform focused on business and employment. This service 
was chosen because of its public data about the relation between workers and 
platforms, making possible a segmentation by country. In order to construct a 
professional self-presentation, many Brazilians list themselves on LinkedIn as 
‘employees’ of these companies. This facilitates finding worker profiles linked 
with these global AI platforms. Our investigation found a significant num- 
ber of these workers on the LinkedIn profiles of Appen and Lionbridge. On 
Appen’s LinkedIn profile, Brazilians were the second largest nationality group 
(776 workers, at the time of writing), just behind Americans and followed by 
Filipinos, Indonesians and Indians. On Lionbridge’s LinkedIn profile, 137 
Brazilians were listed, representing the fourteenth largest nationality group. 
From these two lists of hundreds of Brazilians, we selected 63 workers that we 
were able to contact through a message on LinkedIn, which limits contacts to 
members that have connections in common. 

This initial observation that Appen and Lionbridge had the highest numbers 
for Brazilian workers was corroborated by a search for online groups that we 
conducted on Facebook. We found Appen and Lionbridge workers’ groups with 
a significant number of members: the two biggest groups had respectively more 
than 4,000 and 1,000 workers. In these groups, we found an intense cycle of con- 
versations with dozens of daily posts. Most of the debates had as their subject 
some specificities of the common experiences of workers at these companies. 

Regarding the objective of analysing work of Brazilians on global AI plat- 
forms from the workers’ point of view, we conducted participant observa- 
tion during the research period on the two biggest groups on this subject on 
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Facebook (Hewson 2014). During this period, we interacted with the groups’ 
members, aiming to map the themes and work practices that they discuss. We 
also had contact with WhatsApp groups advertised on the Facebook groups 
that we analysed. 

Then, we conducted semi-structured interviews with 16 workers through 
messaging applications (Brinkmann 2014). Moreover, we contacted workers 
listed as ‘employees’ in the LinkedIn profiles of Appen and Lionbridge and the 
members of the online groups that we interacted with that manifested interest 
in giving interviews. Finally, we surveyed an additional 15 workers through 
a questionnaire with 15 questions about working conditions. Regarding these 
research efforts, the sample was composed of notes and excerpts of partici- 
pant observation, 16 semi-structured interviews and another survey with 
15 responses. 

Acknowledging the diversity of the corpus, composed of notes about par- 
ticipant observation, survey answers and online interviews, we conducted an 
exploratory analysis aiming to map recurrent issues of this environment. In 
this initial overview, it was possible to identify a list of notable subjects that 
emerged more frequently. We began by describing general aspects of these 
workers’ points of view, and subsequently we divided the findings into five cat- 
egories regarding their relation to workers’ perception: hiring processes; time 
tracking and difficulties proving hours worked; lack of infrastructural reliabil- 
ity; work strategies; and, finally, their definition of work and understanding of 
AI. Allthe categories and their types are intertwined and some ofthem overlap. 
The categorisation proposed in this chapter is intended as an exploratory effort 
that can help in understanding the specificities of these global value chains. 


Analysis 


It is immediately possible to note a significant range of different jobs that are 
performed by these workers. Organised into different ‘projects, these ‘tasks, as 
these specific work activities are called, can be very different types of data pro- 
duction: rating advertisements, correcting intelligent assistants’ responses, cor- 
recting map information, producing personal data, analysing Facebook pages’ 
relation with real businesses, categorising images, responding to surveys, tran- 
scribing, translating, subtitling and recording audio or video, etc. 

During the interviews we analyse how these workers understand advantages 
and disadvantages of this type of work. The main advantages observed are 
related to flexibility, which was the topic most verbalised. Interviewees usually 
presented these activities as a job that they can do in different places at different 
times: ‘I can do it while Pm watching TV, while Pm watching some TV series. 
I prefer it because it is time that I can turn off my brain and still make money, 
said Daiana, a 25-year-old odontology undergraduate student living with her 
parents. Laura, 41 years old, argued that this flexibility helps her ‘work without 
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leaving home so I can take care of my 6-year-old son and do the housework. 
Another frequently mentioned advantage is the payment in dollars, given that 
the exchange rate (at the time of writing) shows an aggressively favourable 
trend, boosting their income. 

The most frequently mentioned disadvantage is the lack of job security, an 
aspect that shapes workers’ overall understanding of their activities. As we will 
see in next sections, this lack of stability encompasses multiple factors such as the 
possibility of sudden termination or non-payment of wages. Geraldo, a 50-year- 
old IT worker, said: “There are some tasks that I cant understand and if my 
answers are classified as wrong, I will lose 50% of my wages or lose my account. 
So, it’s nice money, in dollars, but I don't have security. The rules change a lot 
and we are not always informed’ This sense of continuing instability, combined 
with the low pay, are factors that are linked with the evidence that the majority 
of the workers that we interacted with consider this just a side job. Although 
many of them are unemployed, the compensation that they receive from these 
platforms generally is not enough to cover the workers’ financial needs. 

One important issue in the communication between Brazilian workers in 
Facebook and WhatsApp groups is the selection process that workers must pass 
before being chosen for projects on these platforms. Generally, the selection 
processes consist of submitting a résumé in English and passing tests about 
the projects’ guidelines, also in English. They are seen as tough obstacles to be 
overcome and are perceived as somewhat mysterious: ‘It’s impossible that I am 
the only one that is being rejected all the time, says a man in a Facebook group. 
Another member of the group responded: ‘Dude, I don’t know the criterion 
used or if the choice is just random, but I was accepted at the Mechanical Turk 
when I used a new email from Outlook’ 


Time Tracking and Difficulties Proving Hours Worked: 
‘It’s Annoying to be Accused All the Time’ 


In the online conversations and in the interviews that we conducted, the pay- 
ment process is a subject that seems to be connected with many of the work- 
ers’ practices regarding work for global AI platforms, in line with research on 
AMT in Brazil (Moreschi, Pereira and Cozman 2020). Some platforms, such 
as Appen, require workers to self-report hours worked. Although it would be 
quite reasonable to assume that these platforms have the technological capa- 
bility to monitor work, they delegate at least part of this control to the work- 
ers: each one needs to present how many hours she or he has spent doing the 
tasks monthly. It’s common to encounter workers’ narratives about rejections 
and disagreement about the worked hours. Daiana said ‘[The platforms] are 
very disorganized. They dispute hours of work. They say “no, no, no, accord- 
ing to our system you didn’t work those hours.” And then it’s very difficult for 
you to prove the opposite. Sometimes, I just shut up and accept it, because 
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otherwise, you won't get anything! Regarding this, workers share experiences 
about how to verify the activities completed during the worked hours. In a 
Facebook group, a woman complained: ‘Seems silly, but I do everything to 
meet the quota of five to seven working days a week. And for some reason, 
their crazy system just seems to be picking on me. Because it’s no exaggeration 
when I say that I receive at least two [system warnings] per month, and they are 
NEVER true. It sucks to be accused all the time! 

As the first quote of this section illustrates, there is in these conversations 
and narratives a sense of inevitability, that any dispute concerning these alleged 
false accusations is pointless and can also harm workers’ relations with plat- 
forms. On a Facebook group, a man says: ‘a colleague went through a sim- 
ilar situation in [project name]. He sent several screenshots that proved his 
job completion and they never accepted them, unfortunately. Finally, he was 
removed from [project name] at a certain time? Another male worker sum- 
marises this understanding: ‘Whether you complain or not, it doesn’t matter if 
you have your hours recorded, in the end you'll have to accept what they want? 

Work management has historically been based on the control of working 
time as a fundamental resource for work organisation by companies (Wajcman 
2015). In line with other research on platform labour (Woodcock and Graham 
2019; Van Doorn 2017; Abilio 2020), this reveals an algorithmic management 
of labour that produces meanings as impartial, inevitable and unattainable, like 
a data gaze (Beer 2019). However, this does not mean that workers do not com- 
municate - and organise - among them about tactics around working on global 
AI platforms. 


Lack of Infrastructural Reliability: ‘A Lot of Bugs’ 


Frequent lack of infrastructural reliability shapes the work processes of Brazil- 
ians in the global AI platforms. It is necessary to recognise that these techni- 
cal difficulties can vary from one platform to another and between projects 
on the same platform. During the research process, however, we encountered 
different narratives about problems such as platforms ‘bugs, dysfunctional 
apps, connection losses and difficulties regarding payment. The first layer of 
instability may be observed in the structures of the systems that workers use 
to accomplish their tasks. In the interviews, it was possible to understand that 
these difficulties shape workers’ practices and affect their work capacity. Tarcila, 
a 27-year-old law student, relates: ‘I worked on a Facebook project whose appli- 
cation had a lot of errors, a lot of bugs... [...] I had to analyse 30 ads, and some- 
times I opened the application and I had only 15. [...] And sometimes, I sent 
the screenshot to Appen, and they didn't accept it because I didn't have the 
screenshot time, bla bla bla... A lot confusion? 

In Facebook and WhatsApp groups the narratives about platforms’ ‘bugs’ 
produce a range of different perceptions about work. As Tarcila described, it 


258 Al for Everyone? 


is common to see stories about loss of worked hours. A female worker in a 
Facebook group asks if she can be removed from a project for not working due 
to a bug that blocked access to the classification tool: ‘I havent been able to 
classify ads for days because of the bug. Hence today I received the following 
email: “Hello, You are receiving this message because our records show that 
during the week of [a week], you have not completed the requirements for the 
[name] project” These discussions are marked by expressions of feelings such 
as frustration and sadness. 

Linked with the narratives about lost work hours due to platform bugs is the 
development of tactics to overcome these technical difficulties. These tactics 
represent knowledge of the technical logic of these labour processes, strategies 
based on the workers’ experiences and their communication in online groups. 
In the same post mentioned in the last paragraph, another worker responded 
to the question, giving a suggestion about how to avoid system failures: ‘Girl, 
you have to uninstall the Facebook app from your cell and install this version 
that I will put here. I do this every day (uninstall and install again) because my 
phone updates and there is no way to prevent the update’ As this quote shows, 
in this context many of the conversations about bugs are based on the workers’ 
knowledge and may reference platforms’ manuals and guidelines, but also cross 
into the grey areas in the everyday reality of workers which are not covered by 
these documents. 

These narratives about the frequent inefficiency of the platforms’ systems 
move this topic away from the high-tech image that is generally associated with 
the AI debate. This can be even more evident if we consider the problems of 
structure that are an inherent part of many Brazilian platform workers’ every- 
day reality, in line with the research of Moreschi, Pereira and Cozman (2020). 
Problems such as loss of internet connection and glitches in mobile devices are 
frequent and shape the workers’ practices and communications. These situa- 
tions are relatively common in the groups which we interacted with. 

Lastly, another lack of structural reliability for this planetary labour force 
(Graham and Anwar 2019) is represented by the payment systems. As processes 
may differ significantly from one platform to another, payment constrains the 
workers’ routines and their relations with global AI platforms. In the commu- 
nication analysed in our research, these processes are depicted as complex and 
costly, as a dynamic that involves a variety of platforms and financial meth- 
ods. Therefore, different strategies are elaborated and shared with the objec- 
tive of simplifying the receipt of payment and avoiding financial losses. Daiana 
explains her methods for maximising gains: ‘At Appen, you get paid through 
another platform called Payoneer that charges three dollars per withdrawal. 
If I don't have any bills to pay, I gather a significant amount of money and 
minimize withdrawals. On the Lion[bridge], I can economize using the Husky 
platform [...]. The traditional Brazilian banks charge high taxes to receive for- 
eign money. This platform charges only 3.5 per cent of the amount that you 
will receive. So, for me this is good because banks can charge up to 25 dollars, 
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a significant amount considering the value of my wages? In online groups of 
Brazilian workers, the discussions about these strategies are frequent. In a post 
where a worker asks for help avoiding the high fees of a Brazilian bank, a male 
worker said: ‘Guys, for God’s sake, do not transfer directly to the bank, no! They 
will rob you blind. I have been robbed by all of them [banks]. Best option is the 
Husky [platform]? 


Workers’ Strategies: ‘I Don’t Speak German, but, Like, I Roll With It’ 


As was presented in the previous sections, the production and sharing of labour 
strategies is a practice that shapes the dynamics of these online groups. Many 
of the strategies that are discussed in these groups concern increasing or main- 
taining workers’ earnings. Given the economic crisis, work on these platforms 
may represent an important factor in the economic survival of these workers. 
To exemplify this, one of the WhatsApp groups followed during the research 
period, with more than 150 members, has as its title the sentence ‘Online 
Income in the Crisis. 

In our conversations with workers’ different strategic approaches were men- 
tioned as important ways to increase earnings, such as applying to different 
platforms for different projects. Renata, a 46-year-old translator, explains her 
method: “My strategy to increase the remuneration is trying to combine the 
fulfilment of several small and simple jobs with big jobs of bigger values? 

On this subject, it was possible to observe a tension concerning workers who 
werent qualified to do the jobs they had applied for. Tarcila explains that she 
applies for many projects, and then she tries to deal with the specific knowledge 
necessary to accomplish the tasks: “There is a project that is for those who speak 
German. I don't speak German, but, like, I roll with it... I can try to do this pro- 
ject at home! In contrast, Marta, a 27-year-old PhD student, complains about 
those who dont speak English but apply for projects: “You have to know at least 
basic English, and most of the people don't. They only see an opportunity for 
easy money and their work isn’t of good quality? Daiana explains: “People lie a 
lot on their résumés. Marta highlights online translation as a tool for workers 
that do what she calls a low-quality job: ‘People think that the online transla- 
tor is good enough, so they deliver unsatisfactory work’ In the online groups 
that we followed, this tension is still more evident. It was possible to find a 
significant number of posts where language knowledge emerges as a topic in 
dispute. In a post where a worker asks whether he can do tasks in English just 
using Google Translate, another worker responds: “You can use it, but be care- 
ful because if you want to continue on the project and be renewed, it is better 
not to use it or always try to correct it. If not, in six months they will not renew 
your contract’ In another post with the same theme, another worker contests 
this version: ‘I always used big bro Google, and it’s cool, contract renewed. This 
is an urban legend. In other discussions, workers complain about what they 
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perceive as a sense of superiority and hypocrisy from those that critique peo- 
ple who don't know English. One worker protests: ‘I thought it was a support 
group about questions and not a courtroom... Asking for help doesn’t mean 
that I dort know English or that Pm stupid. You dont know enough about 
people's lives to speculate like this. There are people who don't lose opportuni- 
ties to make others feel like garbage, right?’ Similarly, in another post a worker 
says: ‘Let’s be sincere, a large part of the people who are here don’t know how 
to speak English. So, don’t come and say that if the person doesn't know this or 
that she/he wort be able to pass this test, because I guarantee that everyone has 
used or still uses some form of machine translation’ 

On the one hand, the statements above show the tensions identified by Sori- 
ano and Cabanes (2019) between ‘world class work and ‘proletarianised labour’ 
involving digital labour imaginaries in the Global South. The inequalities and 
struggles of human labour behind AI have a strong geopolitical dimension. To 
understand AI, it is necessary to think about spaces of labour. On the other 
hand, this reveals the material conditions of data production. Why do global IA 
platforms need Venezuelans to train data for self-driving cars (Schmidt 2019)? 
Why are Filipinos required for content moderation labour (Roberts 2019)? AI, 
as a techno political and economic project, is based on these inequalities in the 
most diverse layers of mediation. 

The circulation of workers’ struggles, as stated by Dyer-Witheford (2015), 
does not happen in the same way in all parts of the world. Platformized forms 
of colonisation are not just in terms of data, but how they are produced and cir- 
culated by human beings (Couldry and Mejias 2019). This means understand- 
ing AI colonialism, or how resources are expropriated from people in countries 
of the Global South to endorse platforms based in the Global North and their 
mechanisms of value extraction. 

The tension concerning the usage of online translators in the AI platform 
workers’ practices brings out the subject of data production more evidently 
in our analysis. The greatest part of the public discourse about AI emphasises 
its computational potential, which is generally portrayed as a force that can 
reshape society. While the role of data is positioned as a key element in the AI 
infrastructure, the debate about its production is still a secondary topic, and 
these platforms advertise their training datasets as ‘reliable sources’ produced 
by ‘skilled annotators. 

However, the analysis of workers’ narratives and communications presents 
a more complex scenario, marked by many levels of translations/mediations, 
in the philosophical sense that the Actor-Network Theory gives to these terms 
(Latour 2005). This point is reinforced by authors such as Beer (2019) and Coul- 
dry and Mejias (2019). Computationally, data always represents abstraction of 
real phenomena (Wirth 1985). Critically observed, data used in the AI industry 
cannot be considered natural matter that is captured ‘from the world in neutral 
and objective ways’ (Kitchin 2014, 6). This training data is produced by complex 
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processes that involve many different groups of workers, both high and low 
technologies and various socio-economic conditions. In other words, based on our 
analysis we argue that data produced by AI global industry platforms is shaped 
by these work conditions, as a reality composed of many layers of mediation. 


Definition of Work and Understanding of AI: T Say That I Work 
Improving Top Secret Artificial Intelligence’ 


One of our specific objectives in the exploratory analysis of workers’ commu- 
nications was to understand how they perceive their labour practices in the 
context of the global AI platforms industry. Although this didn't represent a 
central topic in workers’ conversations in the online groups that we interacted 
with, it was possible to encounter some discussions about it. In one post, a 
worker asks other members of the group: ‘Guys, just out of curiosity: When 
someone asks you what your work is, what do you answer? I always say that I 
work for [platform] which is an online company, but I never go deeper... and 
you?’ Another worker responds: ‘I say that I work improving top secret artifi- 
cial intelligence: 

The notions of data, algorithm and AI appeared only sporadically and in the 
context of other discussions, as the last quote exemplifies. We consider it possi- 
ble to sustain that many workers in online groups do not see themselves as part 
of the AI industry, viewing their work in a more practical sense, for example, 
as categorising, evaluating, segmenting or correcting information, behaviour, 
content or ads. In our interviews, the situation was similar, but we were able to 
deepen the discussions. Asked whether she understands the relation between 
her work and AI technologies, Daiana says: ‘For me, it was always very clear. I 
always knew I was doing it to train the companies’ algorithms. They say that. 
They say that our work is essential to improve search engines, to train their 
algorithms. In the survey responses, just one respondent spontaneously cor- 
related his/her work with AI: ‘I think I'm helping artificial intelligence systems 
to assimilate cultural aspects, determining funnels that help to show ads and 
contents to specific user profiles? When they were questioned about how they 
think that their work helps to create or train AI systems, the responses focus on 
the idea of improving algorithms and helping users. 


Conclusion 


In this chapter, we described exploratory research conducted through online 
workers’ groups on Facebook and WhatsApp with the intent to deepen and 
diversify the empirical analyses of the work of Brazilians on global AI plat- 
forms. This research involved a diverse corpus comprised of our notes from 
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participant observation and workers’ survey answers, as well as online semi- 
structured interviews. Although the study covers a limited number of work- 
ers, the results are significant and point to some consistent trends. Based on 
this initial effort, we conclude that the labour dynamics of Brazilians engaged 
in work for global AI platforms are complex and evidence several specifici- 
ties. This initial finding corroborates the hypothesis of a taskification of labour, 
as Casilli (2019) states, and we added the geopolitical dimension, in order to 
affirm that there is no digital labour universalism. Platform labour behind 
global AI companies reveals something deeper in relation to working condi- 
tions in countries like Brazil where gig is the norm and whose economy is 
based on informality. 

As the analysis revealed, the workers’ online communication represents an 
important practice that shapes the way they understand their work activities 
and the way they orient themselves in their interactions with these platforms. 
In other words, the knowledge that is produced and negotiated in these online 
environments shapes the workers’ activities as tactics and strategies. The com- 
munication between workers represents a historical phenomenon associated 
with the labour practices. However, in the context of the platformization of 
labour (and consequently the isolation of the workers) this communication 
represents a key element of what Abilio (2020) termed ‘management of sur- 
vival’, proving that workers aren't unorganisable. Platforms can be considered as 
means of communication and production (Williams 2005). Thus, communica- 
tion helps both in the organisation and control of work and in the organisation 
and strategies of workers. 

Finally, we consider that our research reinforces the idea that the datasets that 
fuel AI models need to be understood in the context of complex global chains 
of digital labour. The fact that computational data is an abstraction that embod- 
ies its conditions of production should prompt us to consider the various layers 
of mediation (some of them presented in our analysis) that these data produc- 
tion systems encompass, especially because these systems ultimately shape AI 
decision-making processes. Approaching the potential and agency of AI with- 
out considering the conditions of data labour, we sustain, represents replicate 
uncritical understandings that depict AI as objective, high-tech computation 
produced by the Global North. We assert that critical AI studies have to consider 
the Global South perspectives, acknowledging that data production for automa- 
tion is not a homogeneous process, neither in relation to workforce composition 
nor to platform structure specificities. The labour dimension, we sustain, is a 
vital component in approaches that aim to be critical about AI. As our explora- 
tory analysis shows, there is more negotiation, conflict and low-tech in the AI 
industry’s Global South workforce than is presented in the global AI platforms’ 
discourse. There is no digital labour universalism nor a homogenous workforce 
regarding heteromation of labour on AI platforms. Rather, there is an AI coloni- 
alism reinforcing North-South inequalities from a platform labour perspective. 
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CHAPTER I5 


Towards Data Justice Unionism? 
A Labour Perspective on AI Governance 


Lina Dencik 


Introduction 


The advent of data-centric technologies has in recent years culminated in the 
hype surrounding ‘Artificial Intelligence’ (AI), widely seen to propel transfor- 
mations across areas of science, government, business and civil society. These 
transformations are often simultaneously touted as enhancing forms of effi- 
ciency and better decision-making at the same time as presenting significant 
societal challenges. The question of jobs and the changing nature of work has 
been one prominent area where Al is said to have dramatic implications. Work- 
ers are subjected to evermore data collection about not just their activities at 
work, but beyond factors related to work. At the same time, machine learning 
systems are using this data to transform how work is being allocated, assessed 
and completed. Often it is these two components - data collection and machine 
learning - that is referred to under the banner of AI (Sanchez-Monedero and 
Dencik 2019). This has a profound impact on workers lives, the nature of jobs 
and the economy. Moreover, the position of labour in relation to AI brings to 
light the social stratifications embedded within and created by the advance- 
ment of Al across social life. AI extends long-standing debates on modes of 
capitalism that significantly shape the circumstances of working people whilst 
limiting their ability to influence decisions that govern their lives. 
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Yet, in advancing governance frameworks that may contend with the chal- 
lenges of data infrastructures and AI, there has been a notable absence of worker 
voices, unions and labour perspectives. Labour concerns have predominantly 
focused on the immediate threat of job losses and changing forms of work, 
but these have rarely been translated into AI governance debates more broadly. 
Instead, we have seen governance frameworks emerge that centre on questions 
of individual rights, data protection, ethics and fairness, privileging citizen 
and consumer rights over workers’ rights. These frameworks tend to engage 
at a level of abstract principles that centre on the nature of technology rather 
than the conditions of injustice in which technology is situated, and struggle 
to account for AI as an outcome of power dynamics and interests that serve to 
shape social relations. The absence of labour perspectives within these debates, 
and voices that extrapolate from workplace struggles to a wider engagement 
with AI in the context of advanced capitalism, is therefore a significant gap in 
the context of what we might refer to as data justice; an understanding of data- 
fication in relation to social justice (Dencik, Hintz and Cabel 2016). 

In this chapter, I argue for the need for ‘data justice unionism, a form of 
social justice unionism that engages with data-centric technologies as firmly 
situated within a workers’ rights agenda and that approaches AI governance as 
informed by the labour movement in solidarity with other social movements. 
I start by briefly outlining how AI relates to issues of labour before going on 
to discuss a range of dominant frameworks for AI governance. These have 
tended to exclude broader labour concerns and often frame what is at stake 
with AI in terms of trade-offs between economic gains and individual rights 
that bypass an engagement with collective rights and more fundamental ques- 
tions about the political economy of AI. I then go on to discuss key issues in 
AI that a labour perspective foregrounds, in the workplace and beyond, before 
situating these in relation to data justice unionism. As a component of social 
justice unionism that argues for unions working in coalition with other 
social movements to advance a more just society, data justice unionism makes 
an explicit connection between digital rights and socio-economic rights and 
contends with AI in the context of labour relations under capitalism. This needs 
to inform much more of current mobilisation efforts around data justice in 
order to, on the one hand, elevate the relevance of the labour movement, and, 
on the other, for AI governance debates to better account for lived experiences 
and actual social struggles. 


Labour and AI 


The implications of the advent of AI for labour and labour relations has gar- 
nered much attention in recent years, building on long-standing debates on the 
transformative potentials of emerging technologies. Whilst some have argued 
that the rapid development of data-centric technologies signal a fundamental 
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shift in the operations of capitalism, others have focused on how these tech- 
nologies entrench or extend particular features of capitalism that significantly 
impact on the nature and experiences of work. It is not the aim to discuss these 
different perspectives in detail here, but it is worth briefly outlining some of 
the ways in which labour concerns manifest in understandings of AI in order 
to understand their significance for AI governance debates. At one level, these 
concerns are often focused on the changing nature of the workplace itself and 
how the implementation of AI systems impact on employment relations 
and working conditions. Algorithmic management of the kind associated with 
the gig economy, for example, is rapidly becoming embedded within larger 
parts of the labour market, stretching across different kinds of workplaces (Kel- 
logg et al. 2020). Devices and tools such as phones, laptops and emails are sub- 
ject to monitoring, whilst data extracted from social networks, shared calendars 
and collaborative working tools are being integrated to gain insights into not 
only professional activities, but also who workers are or what they might do in 
the future. More and more, this is being complemented by sensory and recogni- 
tion tools such as chips and wearables that dictate tasks and assess emotional 
and physical states as part of work performance (for an overview of data-driven 
technology in the workplace see Sanchez-Monedero and Dencik 2019). Based on 
this perpetual generation of data within and beyond the workplace, AI systems 
promise to automate key aspects of the labour process and management tech- 
niques. For some, these developments continue a trajectory of automation that 
has long been seen as a threat to labour in different ways, moving beyond the pro- 
duction process of the industrial era to also include information processing and 
decision-making (Andrejevic 2019). Key concerns have been raised about how 
the use of AI technologies in the workplace might displace jobs, impact on work- 
ers rights and undermine labour power (Moore, Upchurch and Whittaker 2017). 

Moreover, engagement with the AI-labour nexus has provided impetus for 
the on-going debate on the implications of emerging technologies for transfor- 
mations in capitalism. Early accounts of the advent of information and com- 
munication technologies (ICTs), for example, indicated a significant shift in 
the relationship between capital and labour and emphasised value extraction 
outside of production as the principal location of the process of valorisation. 
Notions such as ‘immaterial labour’ and ‘cognitive capitalism’ (Moulier Boutang 
2011) point to an elevated significance of knowledge, information and intel- 
lectual property over labour as traditionally understood in operations of capi- 
tal that, for some, promises new visions of ‘post-capitalism’ (Mason 2016) 
and the possibilities for a ‘world without work (Srnicek and Williams 2016). 
Whilst these accounts have been criticised for lacking sound empirical basis 
and often underestimating the continued centrality of production and extrac- 
tion of value from labour in supply chains (Thompson and Briken 2017), they 
point to particular processes in capitalism that have found resonance in more 
recent accounts of the value of data and the political economy of data-centric 
technologies such as AI. 


270 Al for Everyone? 


In Zuboff’s (2015, 2019) notion of surveillance capitalism, for example, value 
generation relies not on a division of labour, but a division of learning: between 
those who are able to learn and make decisions based on global data flows, 
and those who are (often unknowingly) subject to such analyses and decisions. 
In this model, capital moves from a concern with incorporating labour into the 
market as it did under previous forms of capitalism, to a concern with incor- 
porating private experiences into the market in the form of behavioural data. 
This is an accumulation logic driven by data that aims to predict and modify 
human behaviour as a means to produce revenue and market control. Yet in 
thinking about the value of data and the social relations that emerge from its 
extraction, Sadowski (2019) argues that we need to understand it not just as a 
commodity but as capital that propels new ways of doing business and gover- 
nance in what he sees as the ‘political-economic regime of datafication. Data 
collection is driven by the perpetual cycle of (data) capital accumulation, which 
in turn drives capital to construct and rely upon a universe in which everything 
is made of data. The digital platform is central for this transformation in that 
social practices are reconfigured in such a way that enables the extraction of 
data (Couldry and Mejias 2018). Data in this context serves to sustain an eco- 
nomic process that relies on capturing value through expanding the capacity 
for gaining information rather than creating value through production. This 
process does not break with how we might understand capitalism, but rather 
positions datafication as the extension of financialisation and the drive to turn 
everything into a financial asset. The aim is to latch onto circuits of capital and 
consumption for the purposes of rent extraction, whether in monetary form or 
as data (Srnicek 2017; Sadowski 2020). 

The role of labour, under these conditions, is characterised by what Van 
Doorn and Badger (2020) call dual value production’: the monetary value 
produced by the service provided is augmented by the use and speculative 
value of the data produced before, during, and after service provision. As 
they go on to explain, the value of data derives in part from its expected or 
actual practical utility (achieving functional goals and systems optimisation) 
but also from the expectation of data-rich companies to achieve competitive 
advantage and thereby attracting venture capital and higher financial valua- 
tions. AI is part of a suite of complex technologies that have been designed to 
extend and empower capital’s abilities of assetisation, extraction and enclosure 
(Sadowski 2020) and that is rooted in the positive feedback loop between a 
data-producing labour process and algorithmic systems that self-optimise as 
they analyse this data (Van Doorn and Badger 2020). For Wark (2019), this 
constitutes a power shift from the owners of the means of production to the 
owners of the vectors along which information is gathered and used, what 
Wark describes as the ‘vectorialist class. This class controls the patents, the 
brands, the trademarks, the copyrights, and most importantly the logistics 
of the information vector. As such, Wark contends, whilst a capitalist class 
owns the means of production, the means of organising labour, a vectorialist 
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class owns the means of organising the means of production. Importantly, this 
does not necessarily make away with the exploitation of labour in value chains. 
As Srnicek (2020) has pointed out, AI systems rely not just on vast amounts of 
data, but on significant computational power and control over labour to drive 
monopolisation. We have a growing economy based on what Gray and Suri 
(2019, ix) refer to as ‘ghost work’: a new digital assembly line that aggregates 
the collective input of distributed workers, ships pieces of projects rather than 
products, and operates continuously across a host of economic sectors in order 
for AI systems to function. 

The implications of AI for labour therefore extend from the workplace to 
the reorganisation of employment through to the operations of capital upon 
which AI depends and advances. The use of AI in automated hiring systems, 
performance assessment tools, scheduling, and other forms of algorithmic man- 
agement in the workplace (platforms or otherwise) intersect with broader trans- 
formations in the economy and dynamics of capitalism in which developments 
in AI are embedded. These different concerns highlight the many complex and 
intricate ways AI impacts on the experiences of working people, the way their 
work is organised and how it is valued, and their ability to influence decisions 
that govern their lives. Yet, as I will go on to outline below, workers’ voices and 
union perspectives have been notably absent from AI governance debates that 
have instead overwhelmingly championed liberal frameworks based on citizen 
and consumer rights. If we are to contend with AI in relation to the advance- 
ment of a more just society, then such frameworks are insufficient. 


Governing AI 


The advent of AI has sustained much discussion about what is actually at stake 
with the growing datafication of social life. Whilst there is widespread recogni- 
tion that the rapid development and deployment of data-centric technologies 
has significant transformative implications, the question as to what these are 
and how they should be addressed is still up for grabs. Gangadharan (2019) has 
provided a useful overview of different frameworks for data governance that 
highlights some of the dominant ways in which AI and data-driven systems in 
general have been approached in governance debates, including privacy policy, 
data protection, ethics, fairness-in-design and human rights. Elaborating on 
this overview, I argue in this section that mobilisation around the governance 
of emerging technology, particularly AI, has thus far been situated in a digital 
rights and technology-driven agenda that has foregrounded individual rights 
and focused on the nature of the technology itself. Lacking from this agenda 
has been a more substantial engagement with collective rights and the actual 
conditions of injustice and lived experiences of struggle within which technol- 
ogy is embedded. The labour movement has an important role to play bringing 
such a perspective forward within the AI governance space. 
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Initial concerns over the mass generation and analysis of data collection have 
tended to highlight issues of surveillance and privacy, prominent in public 
debate particularly in the immediate aftermath of the Snowden leaks first pub- 
lished in 2013 (Hintz, Dencik and Wahl-Jorgensen 2018). In part, these events 
made clear the limitations of existing legislation around privacy, and the need 
for more oversight in the handling and processing of data by different actors. 
This saw the flourishing of a range of technology and policy initiatives aimed at 
restricting data gathering, such as the development of privacy-enhancing tools, 
mainstreaming the use of encryption and lobbying around anti-surveillance 
issues (Dencik, Hintz and Cable 2016). These have advanced important reper- 
toires for resistance that directly challenge the power relations of data-driven 
surveillance and have provided avenues for individuals to manage aspects of 
their digital engagement. However, the advancement of technological self- 
defence as a governance frame is also limited by the onus on the individual user 
to protect their own privacy. As Ruppert, Isin and Bigo (2017) describe it, many 
accounts of data politics are premised on an ontology of ‘hyper-individualisnY 
that nurtures a suggestion that ‘ultimately it is up to you to change your behav- 
iour to protect yourself from the dark forces of the internet. 

In translating some of the concerns of anti-surveillance resistance into regu- 
lation, the protection of personal data has been a particularly noteworthy frame 
for governance, such as the approach to the General Data Protection Regula- 
tion (GDPR) adopted by the EU in 2018. The premise is that individuals should 
be able to claim some rights with regards to information collected about their 
person, and that collecting such information requires some form of consent. In 
this sense, it privileges the individual data subject and understands the protec- 
tion of personal data as distinct from, but complementary to, individual pri- 
vacy. The GDPR is relatively broad in scope but it is worth noting that issues 
pertaining particularly to the workplace and the processing of data on workers 
were excluded from this regulation in its final stages (Colclough 2020). Rather, 
the GDPR predominantly favours an understanding of data subjects as indi- 
vidual citizens and consumers that are afforded certain rights about their abil- 
ity to access, challenge and limit data collected about their person by private 
companies and parts of the public sector. 

Although the GDPR has paved the way for engaging with data-centric tech- 
nology in a broader sense, questions remain about both its scope and enforce- 
ability. Perhaps in part as a response, much attention and resources have been 
dedicated to advancing ‘data ethics’ and ‘AI ethics’ in recent years as alternative 
and complimentary governance frameworks. This field has engaged a range 
of different streams of thought and practice, some of which continue a long- 
standing tradition of computer ethics while changing the level of abstraction of 
ethical enquiries from an information-centric to a data-centric one (Floridi and 
Taddeo 2016). That is, the focus shifts from a concern with how to treat infor- 
mation as an input and output of computing to a focus on how people access, 
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analyse and manage data in particular, not necessarily engaging any specific 
technology but what digital technology manipulates (Taylor and Dencik 2020). 
Often this has privileged concerns with the responsible handling of data that 
considers risks to privacy, forms of discrimination and abuse, ensuring trans- 
parency and accountability. In translating this into practice, we have seen the 
proliferation of various initiatives across industry, government and civil society 
framed under ‘ethics’ that set out different guidelines and procedures that attend 
to the development, handling and deployment of data-centric technologies, 
particularly AI. Government initiatives such as the UK’s Centre for Data Eth- 
ics and Innovation and the establishment of high-level expert groups on ethics 
within the EU have advanced some avenues for outlining ethical concerns in 
relation to technology, whereas civil society actors have turned to data ethics 
as a way to advance data developments ‘for good’ across a range of contexts. 
Of particular note has been the active engagement by the technology sector 
itself in this governance frame, swiftly setting up associations, creating guide- 
lines and codes for the responsible handling of technological innovation. An 
early offering came in the form of the Partnership on Artificial Intelligence to 
Benefit People and Society set up by Amazon, Google, Facebook, IBM and 
Microsoft in 2016 as a non-profit organisation to advance ‘best practices 
and public understanding: Most of these companies have also subsequently 
attempted to set up their own ethics boards, sometimes in partnership with 
academics, with varying degrees of success (Naughton 2019). 

While a focus on data and AI ethics has foregrounded some prominent con- 
cerns about data collection and use in a way that shifts the onus of responsibil- 
ity onto developers and the data controller, it is not clear that these initiatives 
have resulted in any real intervention. Government entities have predomi- 
nantly been set up as nominal oversight bodies without any real teeth to inter- 
fere, leaving civil society actors having to levy at the abstract level of principles 
and rely on the goodwill of the industry to uphold them. Corporate data ethics 
initiatives, meanwhile, have focused on ‘micro-ethics, an orientation around 
the individual practitioner, and an emphasis on compliance that avoids any 
fundamental engagement with the bottom line or premise (Taylor and Dencik 
2020). In some instances, this has led to accusations of ‘ethics-washing’ (Wagner 
2018), allowing for technology companies to engage with public concerns 
about their activities, while continuing to avoid regulation or any major chal- 
lenge to the business models that sustain them. Moreover, by actively cap- 
turing the ethics space, the very players who are creating, developing and 
directly profiting from these technologies have also been the ones dictating 
the terms upon which we are to understand both the nature of problems and 
what might be suitable responses. Unsurprisingly, therefore, the application of 
ethical frameworks within the technology sector has tended to concern itself 
with the actual data-sets or algorithms themselves, positing that the causes of 
harms that emerge from AI can be traced to errors’ or ‘bias’ in the design and 
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application; causes that essentially have technological solutions, preferably 
through further data collection and algorithmic sophistication. We see this for 
example with the growing industry that now concerns itself with ‘fairness’ in 
the design of systems, creating more inclusive data-sets and algorithms that 
can account for more diverse experiences, or the development of ‘bias mitiga- 
tion’ tools (Zelevansky 2019). Such projects have drawn attention to some of 
the contentious assumptions that are embedded in the design of technological 
systems, but have also been accused of advancing technical fixes that serve to 
legitimise the industry (Gangadharan and Niklas 2019). 

The growing debate surrounding ethical challenges and the bias of algorith- 
mic processes has helped spur on an engagement with data-driven technologies 
as socio-technical systems that have an impact on people’s lives. Some of this is 
evident in emerging forms of regulation on AI, for example, the emphasis on 
“Trustworthy AT and a risk-based approach to minimising harms in AI systems 
at EU level (Niklas and Dencik 2020). It has also been prevalent in discussions 
on the future of work, for example, by attending to the ways in which hiring 
systems or other parts of automation in human resources might discriminate 
against particular groups (Ajunwa 2019; Graham et al. 2020). However, con- 
cerns about ethics washing and the tendency towards technical fixes have led to 
calls to centre rights, and particularly human rights, more firmly within these 
discussions. Drawing on human rights legislation in AI governance debates 
goes beyond issues of privacy and the protection of personal data whilst pro- 
viding a sturdier point of reference than abstract principles of ethics and fair- 
ness. Using international human rights as a frame in relation to the governance 
of AI details the specificity of potential harms by linking them to particular 
rights, such as the right to freedom of association or the right to a fair trial, that 
can apply to different parts of social life (HRBDT 2020). These assertions of 
rights can help inform impact assessments, for example, when new AI systems 
are being developed or deployed (Jorgensen et al. 2019; Jansen 2020). By rely- 
ing on universal terms of reference, a human rights framework is also effective 
for advocacy as an internationally recognised agreement, however much this 
may not play out in practice. A recent court case brought forward by NGOs in 
the Netherlands, for example, to challenge the use of data-centric technology in 
the welfare sector won on the basis that it was considered an infringement on 
human rights and supported on-going efforts by the human rights community 
to demand assessments of AI systems beyond the required initial data protec- 
tion impact assessment (Toh 2020). 

Governing AI from a human rights perspective can therefore provide an 
avenue for a more holistic engagement with data-driven systems that consid- 
ers a broad range of rights that pertain to people’s lives. However, the notion of 
international human rights has historically struggled to translate into success- 
ful concrete action, often seen to be at the whims of geopolitical concerns and 
international relations. Moreover, as a framework, it has traditionally centred 
on the individual and civil and political rights in a way that has struggled to 
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account for collective rights and that has tended to neglect social and economic 
rights (Alston 2005). In general, in line with how governance debates on AI 
have predominantly been approached, there is a lack of political mobilisation 
that can contend with the power relations that are inherent in the advancement 
of Al and that engages with datafication as a political economic regime. The 
absence of labour concerns is an important aspect in this respect. In part, this 
is a result of a deliberate exclusion of worker voices in governance debates, both 
in how they have been organised as well as in terms of institutional structures 
surrounding AI governance. It is noteworthy how even discussions on AI in 
relation to the future of work have been advanced around industry and citizen 
concerns, but with an absence of unions or other worker associations. At the 
same time, the lack of labour perspectives in relation to AI governance is also 
indicative of a labour movement that has been slow to engage with questions of 
data and data-centric technologies on a societal scale. As I will go on to argue, 
unions and labour activists have predominantly (understandably) focused on 
immediate concerns regarding the changing nature of work and the workplace, 
particularly with the advent of the gig economy and automation. This focus has 
brought to light some significant issues with regards to the power of technol- 
ogy companies in setting the terms of work and workers’ worth, but it has not 
translated into mobilisation efforts around AI governance, the way in which 
AI positions labour in relation to capital, and how this informs the advance- 
ment of social justice. Instead, as I will go on to discuss below, other actors and 
communities that could benefit from alliances with the labour movement have 
driven mobilisation around this kind of data justice. 


Data Justice and Labour 


Privileging a concern with social justice in relation to datafication, a framework 
of data justice, is part of a notable shift in the framing and understanding of 
what is at stake with the growing development and use of data-centric technolo- 
gies such as AI. In part a response to the rather limited interpretations that have 
informed governance debates thus far, data justice advances a research agenda 
that seeks to change the terms of the debate, situating data in relation to struc- 
tural inequalities and histories of domination (Dencik, Jansen and Metcalfe 
2018). This has, in some interpretations, led to articulations of principles to 
underpin data governance that can better account for such inequalities (Heeks 
2017; Taylor 2017), or practices in the handling of data that make asymmetries 
in the representation and power of data explicit (Johnson 2018). In other inter- 
pretations, conceptions of justice have been foregrounded in the development 
of design, calling for more participatory processes that involve communities 
to build alternative infrastructures that empower rather than oppress mar- 
ginalised groups (Costanza-Chock 2018). This is in line with a more general 
recognition of the need to shift what voices are centred in any understanding 
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of what is at stake with datafication and challenge the current constitution of 
the decision-making table. Gangadharan and Niklas (2019), for example, have 
made the case for decentering’ technology in data justice debates and instead 
situate technology within systemic forms of oppression, meaning that harms 
that emerge from data-driven systems need to be articulated by those who are 
predominantly impacted and understand the history of such oppression. 

We have seen some of these tenets of data justice debates translate into dif- 
ferent forms of activism and campaigning. The Center for Media Justice in 
the United States, for example, have created a Data Justice Lab dedicated to 
thinking through ways to bridge research, data, and movement work relat- 
ing to issues like surveillance, carceral tools, internet rights and censorship. 
The Detroit Digital Justice Coalition has worked with local residents to iden- 
tify harms that emerge through the collection of data by public institutions, 
situating these in the context of on-going criminalisation and surveillance of 
low-income communities, people of colour and other targeted groups. In some 
instances, these activities have foregrounded a politics of refusal (Gangadharan 
2019) that advance an abolitionist agenda as articulated by groups such as 
the StopLAPD Spying Coalition and the Data for Black Lives initiative. Here, the 
focus is not to make technologies more efficient, but rather to recognise how 
technology has meaning and impact in relation to the inequalities manifested 
in capitalist exploitation and a history of state violence. The call is to divest 
resources into oppressive data systems and to ‘abolish big data that is used to 
measure and profile people, and instead reinvest in communities (Benjamin 
2019; Crooks 2019). In the context of environmentalism, the Environmental 
Data & Governance Initiative (EDGI) has preserved vulnerable scientific data 
in the aftermath of the US election of Trump in 2016, and in the process devel- 
oped an ‘environmental data justice’ framework that considers the politics, gen- 
eration, ownership and uses of environmental data. Similarly, in the context of 
municipalities, efforts to engage citizens in the control over urban public data 
have been central to the ‘Roadmap to Technological Sovereignty’ advanced in 
cities such as Barcelona, outlining ways to challenge the monopolisation of data 
by a few corporate platforms. These efforts tend to focus on forms of govern- 
ance that include formats such as data trusts or data commons and that allow 
for platforms to be managed by the city itself (Tieman 2017; Fuster 2017). 

Concerns with data justice therefore translate into a range of different debates 
and practices that find expression across areas of society. Yet whilst these activ- 
ities speak to shared interests towards addressing inequalities, redistribution 
and conditions of injustice, labour concerns have often been on the margins 
of these efforts or have been pursued in siloes. There has been a considerable 
effort to address the issue of potential job losses in the face of automation, for 
example, with unions pushing for more avenues to pursue reskilling within 
jobs, changing union structures to accommodate for non-trade or non-sec- 
tor specific memberships, and advocating for more support for transitioning 


Towards Data Justice Unionism? 277 


within workplaces (Colclough 2020). Concerns with job losses have also mobi- 
lised greater support for some form of universal basic income or other kinds 
of safety nets for workers who are displaced by automation (Standing 2019). 
More recently, there has been a growing focus on how technologies such as AI 
impact not just displacement, but the quality of work. This includes efforts to 
apply the GDPR in the workplace as a way to address issues of labour protec- 
tion (Aloisi and Gramano 2019) and the potential for collective rights to form a 
greater part ofthe Aland data governance debate. De Stefano (2018), for exam- 
ple, has argued for the need for a ‘human-in-command’ approach that would 
involve collective regulation and social partners in governing automation and 
the impact of technology at the workplace. Similarly, some unions are pushing 
for ‘new technology agreements’ (NTAs) to form part of collective bargaining 
agreements in workplaces that have union representation. Under the terms of 
these, new technology will only be introduced with the agreement of the trade 
union, and if the employer agrees to reinvest any cost savings to provide new 
jobs elsewhere in the organisation (Cole 2019). 

Furthermore, spearheaded by smaller and independent unions, the labour 
movement has been increasingly active in the area of platform labour and the 
gig economy (not all of which deploy AI), focusing particularly on the nature 
of these platforms as employers. Unions such as the Independent Workers of 
Great Britain (IWGB) and what is now the App Drivers and Couriers Union 
(ADCU) have successfully challenged the status of gig workers, such as those 
driving for Uber, as self-employed rather than as employees. Out of these strug- 
gles, there has also been a growing engagement with the collection and use of 
data by these platforms to manage or direct workers employed by them. Worker 
Info Exchange, for example, a non-governmental organisation that grew out of 
organising Uber drivers within the IWGB, explicitly concerns itself with ‘data 
rights’ and the ability for workers to access data collected about them as a way 
to increase transparency about their management. In parallel to this, a growing 
mobilisation effort has formed around alternative models of platform labour 
that draws inspiration from the cooperativist movement. Platform cooperativ- 
ism as an idea and practice has, in the space of a few years, grown globally as a 
response to the dominance of platform capitalism. Under this model, platforms 
are generally based on decentralised forms of governance in which workers 
themselves own the platform and/or set the terms for how it should be run. 
Importantly, these have sometimes been established with the direct support of 
labour unions and have been an avenue through which to engage the labour 
movement more directly in data debates. As Scholz (2017) has argued, platform 
cooperativism should not be considered a technological solution but a ‘mind- 
set’ that includes technological, cultural, political and social changes, bringing 
together different actors and stakeholders. 

The growing arena of data justice therefore has much scope to incorporate 
labour concerns in how to articulate both what is at stake with datafication 


278 Al for Everyone? 


as well as possible responses. In terms of mobilisation, the challenge remains 
how to integrate issues of data-centric technologies like AI into a broader 
understanding of the place of technology in advancing social justice for all. As 
I go on to argue below, this needs greater cooperation and solidarity between 
the labour movement and other social movements in order to strengthen 
and advance the kind of political engagement with AI governance that a data 
justice agenda demands. 


Towards Data Justice Unionism 


As Ihave argued so far, dominant governance frameworks relating to data and AI 
have overwhelmingly responded to concerns with implications for people as 
citizens and consumers over and above people as workers. Moreover, they have 
predominantly followed a liberal orientation that has centred on individual 
rights and ethics. This is perhaps unsurprising considering that much of the 
mobilisation efforts engaged in questions of data and AI have been those stem- 
ming from digital rights and civil liberties concerns. Whilst this has mobilised a 
number of key issues that have been translated into important legislation, par- 
ticularly in Europe, such as the GDPR and more recently, the White Paper on EU 
Strategy for AI, such frameworks have some important limitations for engaging 
with the broader implications of the turn to data infrastructures across social 
life. The growing activities surrounding data justice have broadened and shifted 
the terms of engagement in ways that seek to address some of these limitations. 
Yet labour concerns regarding AI have often been pursued separately from these 
activities. This is a challenge for broad political mobilisation as the labour move- 
ment has historically played a significant role in connecting transformations in 
work to broader questions of society that have relevance for the governance of 
data and AI. In this final section, I therefore make the case for data justice union- 
ism to be considered as a part of social justice unionism focused on engaging 
labour perspectives in the debate on AI governance, including a concern with 
the interests driving datafication, the forms of social and economic organisation 
that enable them, and how they might be challenged. 

Social justice unionism has become an increasingly popular approach within 
the labour movement and advocates for unions to collaborate with social 
movements in order to work towards wider goals and the resolution of work- 
place issues. The argument is that unions should accept the reality that there are 
multiple forms of oppression and that they should work with groups in coali- 
tions to challenge them (Healy et al. 2004; Dencik and Wilkin 2015). This often 
means an emphasis on more networked and informal relationships between 
individuals, groups and organisations that combine to undertake forms of 
collective action. A prominent example is the protests in Seattle in 1999 that 
brought together a diverse array of social movement groups with trade unions 
to protest against the specifics of the WTO proposals but also against the 
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growth of corporate power and the destruction of democracy, a much broader 
theme uniting these movements and providing grounds for a coalition to be 
built on labour-related interests (Wilkin 2000). More recently, the Occupy 
movement brought together union organisers with a range of different 
social movement groups to draw attention to uneven wealth distribution and 
income inequality that formed grounds for demands of a living wage and job 
securityin precarious sectors historically neglected by mainstream trade unions 
(Dencik and Wilkin 2015). These kinds of mobilisations focus on improving 
the lives of working people through engaging with class-wide or social justice 
demands, which include traditional ‘bread and butter’ issues, but are not lim- 
ited to them (Behrent 2015). 

Social justice unionism therefore resonates with the argument that unions 
need to present a picture of a good society that can be built through coopera- 
tion, solidarity and mutual aid alongside other progressive social movements. 
This understanding of unionism has gained particular relevance in light of 
declining labour power coupled with the nature of the global challenges con- 
fronting working people. Climate change, for example, presents a complex 
challenge for the labour movement that cannot be fought along traditional 
lines. The recent push for a Green New Deal and a Just Transition is in part an 
attempt to foster new alliances between movements concerned with labour and 
the environment. This includes unions and campaigners teaming up to advance 
concrete alternatives to a fossil-fuel-based economy, while advocating that 
the government take action. Indeed, Bergfeld (2019) has argued that what is 
needed is a kind of ‘climate-justice unionism’ to address the intertwined social 
and ecological crises in a holistic way. Such an approach would use the organi- 
sational and institutional leverage of unions to rebuild workers’ power at the 
workplace and at company level to regulate from below, whilst at sectoral level 
use collective agreements to refit companies, with the goal of reducing carbon 
emissions and enhancing labour standards. Importantly, climate-justice union- 
ism would involve organising ‘the whole worker’ (McAlevey 2016) in which issues 
are not only rooted in workplaces but also in communities and society, such as the 
disproportionate impacts of exposure to and taxation of CO, emissions. 

A concern with data justice in this context provides a further component 
that needs to be part of these efforts to address issues confronting working 
people within and beyond the workplace, privileging a view of unions as 
working in solidarity with other groups. The engagement with questions of 
data from a social justice perspective cannot be confined to digital rights, civil 
liberties or technologists, but requires a coalition of individuals, groups and 
movements. Unions have an important role to play in this respect, not just by 
explicitly connecting digital rights to social and economic rights, but perhaps 
more importantly by articulating concerns that are rooted in people’s lived 
experiences of AI. This can help mobilise around actual and on-going social 
struggles informed by those who are the most impacted as a key component of 
current data justice debates. Furthermore, unions can leverage power within 
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the workplace to address the deployment of AI systems that can inform gov- 
ernance debates around AI more broadly. McQuillan (2020), for example, has 
advocated for workers and peoples councils to advance situated knowledge 
as a form of interference in relation to AI, drawing on the social histories of 
workplaces and communities. Unions might also help to organise workers 
within technology companies as pursued by groups such as the Tech Workers 
Coalition that sees labour organising as a way of advancing solidarity between 
software engineers and social justice movements to undermine the develop- 
ment of harmful technologies. More broadly, data justice unionism provides 
an avenue for mobilising around AI that engages with the political economy 
upon which its advancement relies. By attending to the operations of capital 
in datafication and its positioning of labour, we are forced to move away from 
a focus on the responsible handling of data or to turn to the realm of moral 
conscience or market solutions as governance responses. Instead, we need 
to contend with the actual conditions of injustice that shape contemporary 
social relations, how AI shifts dynamics of power and approach questions of 
technology as part of alternative visions for how society should be organised. 
This requires coordinated efforts between the labour movement and other 
social movements. 


Conclusion 


The advent of datafication has culminated in recent discussions on AI, bringing 
to light the significant ways in which data-centric technologies are intersect- 
ing with various aspects of social life. A particular area of concern is the way 
labour relations are transforming with the growing development of AI. This 
has often focused on the risk of job losses to automation and the changing 
nature of the workplace, both in standard and non-standard employment. It 
has also incorporated an analysis of the way labour, often side-lined or made 
invisible, is central to sustaining AI systems at the same time as the mode of 
capital advanced by AI undermines labour power by extending and empower- 
ing capital’s abilities of assetisation, extraction and enclosure. Yet in mobilis- 
ing governance frames to contend with datafication and AI, there has been a 
noticeable absence of workers’ voices and labour concerns. Instead, dominant 
frameworks of AI governance have tended to focus on citizen and consumer 
rights that have centred on the individual and on the ethical considerations 
that need to inform design and deployment. Labour concerns, meanwhile, have 
been pursued in separate arenas that have tended to focus on specific aspects 
of work and the workplace, but that have often not connected with broader 
debates on data. As AI comes to have increasing significance for how society is 
organised, there is a need to foster greater cooperation between different move- 
ments and groups to engage with data justice in a meaningful way. Unions can 
benefit from a more holistic form of organising that extrapolates workplace 
issues into society in order to gain relevance and advance the interests of their 
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membership. Engaging with data issues needs to be part of that organising. At 
the same time, unions bring particular leverage to existing efforts to advance 
social justice concerns in the context of AI by privileging lived experiences and 
foregrounding collective social and economic rights. Data justice unionism, 
therefore, is a way of pointing to the potential for a broader political mobili- 
sation around the role of AI in society that involves the efforts and voices of 
actual working people. Such a mobilisation is urgently needed if we are to con- 
tend with the shifting power dynamics that are being advanced by the growing 
reliance on AI in our lives. 
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Al FOR EVERYONE? 


e are entering a new era of technological determinism and 
N ‘eons in which governments and business actors are 
seeking data-driven change, assuming that Artificial Intelligence 
is now inevitable and ubiquitous. But we have not even started asking the 
right questions, let alone developed an understanding of the consequences. 
Urgently needed is debate that asks and answers fundamental questions 
about power. 
This book brings together critical interrogations of what constitutes 
AI, its impact and its inequalities in order to offer an analysis of what it 
means for AI to deliver benefits for everyone. The book is structured in 
three parts: Part 1, AI: Humans vs. Machines, presents critical perspectives 
on human-machine dualism. Part 2, Discourses and Myths about AI, 
excavates metaphors and policies to ask normative questions about what 
is ‘desirable’ AI and what conditions make this possible. Part 3, AI Power 
and Inequalities, discusses how the implementation of AI creates important 
challenges that urgently need to be addressed. 
Bringing together scholars from diverse disciplinary backgrounds and 
regional contexts, this book offers a vital intervention on one of the most 
hyped concepts of our times. 
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